13 KiB
Table of Modules
- Configuration
- Device
- Buffer
- Inference
- Calbin (STILL UNDER DEVELOPMENT)
- Memory Management
- Error Handling
Configuration
This section describes configure function of calculet runtime programming interface.
configure
CalrtError_e configure(CalculetDevice *vDev, Calbin *pParser)
configure calbin to device
Parameter
vDev -- calculet device pointer, including VirtualDevice and CalrtDevice
Return
CalrtSuccess, CalrtErrorInvalidConfiguration
Description
return CalrtSuccess if configure successfully.
Device
This section describes device control functions of calculet runtime under CalrtDevice object.
Scan Device
static std::optional<std::vector<CalrtDeviceId_s>> scanDevice()
scan valid calculet device
Return
Description
if success, a list of Calculet device ids would be returned. It does matter to construct runtime device objects to control physical device.
Create Device
static std::unique_ptr<CalrtDevice> CreateDevice(const CalrtDeviceId_s &id)
create runtime device object
Parameter
id -- a calculet device id.
Return
std::unique_ptr<CalrtDevice>
a valid runtime device unique pointer for controlling physical device
Description
Argument id must be obtained by scanDevice() which guarentee a number of valid calculet device ids are captured. You may choose to create your own device id by initialize struct CalrtDeviceId_s. But it may be failed to construct runtime device object.
Trigger Device Execution
void StartJob(uint64_t jobAddr, uint64_t jobSize) const
trigger device execution
Parameter
jobAddr -- program start memory address in device.
jobSize -- program byte size.
Description
Let device to execute a program at specific memory address.
Wait Device Execution Finish
void WaitJobDone(std::atomic_bool& shutdown) const
void WaitJobDone(std::atomic_bool& shutdown, uint64_t& jobEndInfo) const
wait untill device execution finished and get data back to output buffer.
Parameter
shutdown -- indicate if core-engine is shutdown.
jobEndInfo -- a finished job id.
Description
A blocking function used to get data back to host memory when device finishes corresponding job.
Reset Device
void Reset()
reset device, including memory allocation and firmware to factory setting.
Update Firmware
void UpdateFirmware()
STILL UNDER DEVELOPMENT
update device firmware.
Get Firmware Information
CalrtError_e GetFirmwareInfo()
STILL UNDER DEVELOPMENT
Get Device Information
const char *GetDevInfo()
Get Device Information, including memory usage, core frequency, etc.
Get Device Execution Status
CalrtError_e GetDeviceStatus()
Get Device Statues
Return
CalrtSuccess, CalrtErrorDeviceBusy
Buffer
This section describes buffer function of calculet runtime programming object. It is used for inference, carrying valid data.
Create Buffer
std::unique_ptr<CalrtInputBuf> createInputBuf(const CalbinModel &model)std::unique_ptr<CalrtOutputBuf> createOutputBuf(const CalbinModel &model)
create input or output buffer
Parameter
model -- CalbinModel obtained by parsing calbin file
Return
std::unique_ptr<CalrtInputBuf> runtime input buffer object pointer
std::unique_ptr<CalrtOutputBuf> runtime output buffer object pointer
Get Buffer Information
const CalrtBufferInfo_s& GetBufferInfo() const
Get i/o buffer information
Return
Tensors Information
std::vector<CalrtTensor>& GetTensors()
get i/o buffer tensor reference information and allow to modify it.
Return
Description
With CalrtTensor.GetDataPtr(), it return a char * for input data or get output data.
Ping-Pong Memory
std::vector<CalrtDevBuf_s>& GetTensorMemInfo()
get i/o buffer ping-pong address
Return
Get LLM info
const CalbinLLM_s &GetLLM()
Get max batch size and max sequence length
Return
Note
Only input buffer has this member function. It is NULL if running CNN model.
Get and Modify Current Hyperparameter
ModelHyperParameters_s &GetCurHyperParam()
Get and set current LLM needed hyperparameter
Return
Note
Only input buffer has this member function. It is NULL if running CNN model.
Inference
This section describes inference function of calculet runtime programming interface.
void infer(std::unique_ptr<VirtualDevice> &vDev, CalbinModel *model, CalrtInputBuf* inputBuffer, CalrtOutputBuf* outputBuffer);
Parameter
vDev -- VirtualDevice virtual device manages one or multiple physical device.
model -- CalbinModel try to infer specific model
inputBuffer -- CalrtInputBuf runtime input buffer pointer
outputBuffer -- CalrtOutputBuf runtime output buffer pointer
Description
Non-blocking function used to infer specific model with selected device or multiple devices. use outputBuffer to get inference result.
Calbin
This section describes calbin class part of calculet runtime programming.
Create Calbin
static std::unique_ptr<Calbin> CreateParser(const std::string &path)
Construct an unique pointer of Calbin object
Parameter
path -- location of calbin file
Get Calbin
CalrtCalbin& GetCalbin()
Get a reference of CalrtCalbin data struct
Get model
std::optional<CalbinModel> GetModelByName(const char *modelName)
std::vector<CalbinModel> GetAllModels()
Both member functions can provide a copy of the current model descripted in calbin file. Difference is one for the specific one and the other one is for all CalbinModel data structure.
Memory Management
This section describes CalrtDevice member functions of memory management.
Register Device Memory
CalrtError_e RegisterDram(uint64_t startAddr, uint64_t size)
register(reserve) device DRAM
CalrtError_e RegisterSyncUnit(uint64_t startAddr, uint64_t size)
register(reserve) sync unit
CalrtError_e RegisterSramBuf(uint64_t startAddr, uint64_t size)
register(reserve) SRAM
Parameter
startAddr -- reserved memory start address
size -- reserved byte size
Return
CalrtSuccess, CalrtErrorMemoryAlreadyRegistered
Usage
/*CalrtDevice pointer*/ device_ptr->RegisterDram(0x1000, 32);
Allocate Device Memory
uint64_t CreateBuf(uint64_t size, uint32_t alignLog2Byte=0)
create DRAM buffer
uint64_t CreateSramBuf(uint64_t size, uint32_t alignLog2Byte=0)
create SRAM buffer
uint64_t ApplySyncUnit()
apply a sync unit
Parameter
size -- bytes size.
alignLog2Byte -- align to specific byte size by 2^n. E.G. 64 byte align. then n is 6.
Return
uint64_t buffer start address.
Note
if fail to allocate memory, then program would be abort.
Usage
/*CalrtDevice pointer*/ device_ptr->CreateBuf(32, 20) // allocate 32 byte memory aligned to 1MB on DRAM
Copy from Host Memory to Device Memory
void WriteToDevice(void *srcAddr, const uint64_t devAddr, const uint64_t size)
coyp host memory to device memory
Parameter
srcAddr -- host side buffer start address.
devAddr -- device side memory start address.
size -- buffer byte size.
Copy from Device Memory to Host Memory
void ReadFromDevice(void *srcAddr, const uint64_t devAddr, const uint64_t size)
Parameter
srcAddr -- host side buffer start address.
devAddr -- device side memory start address.
size -- buffer byte size.
Write Device Register
void WriteReg(const uint64_t regAddr, uint32_t data)
write data to register by address
Parameter
regAddr -- register address.
data -- input data
Read Device Register
void ReadReg(const uint64_t regAddr, uint32_t &data)
read data from register by address
Parameter
regAddr -- register address.
data -- output data
Free Device Memory
void FreeBuf(uint64_t addr)
Free DRAM by device address
void FreeSramBuf(uint64_t addr)
Free SRAM by device address
void FreeSyncUnitByAddress(uint64_t addr)
Free sync unity by device address
void FreeSyncUnitByIndex(uint32_t idx)
Free sync unity by index
Error Handling
Calrt Check
check error code if is CalrtSuccess. Otherwise abort program with optional error message.
CALRT_CHECK(CalrtError_e, msg)
Report Runtime Error
Report error code with optional error message and abort program.
REPORT_CALRT_ERROR(CalrtError_e, msg)
Report Runtime Error if
Report error code with optional error message and abort program if condition is false.
REPORT_CALRT_ERROR_IF(CalrtError_e, condition, msg)
Description
work like assert, only report error when condition is false.