# **Table of Modules** # 1. [Configuration](#configuration) 2. [Device](#device) 1. [Scan Device](#scan-device) 2. [Create Device](#create-device) 3. [Trigger Device Execution](#trigger-device-execution) 4. [Wait Device Execution Finish](#wait-device-execution-finish) 5. [Reset Device](#reset-device) 6. [Update Firmware](#update-firmware) 7. [Get Device Firmware information](#get-firmware-information) 8. [Get Device Information](#get-device-information) 9. [Get Device Execution Status](#get-device-execution-status) 3. [Buffer](#buffer) 1. [Create Buffer](#create-buffer) 2. [Get Buffer Information](#get-buffer-information) 3. [Get and Modify Tensors Information](#tensors-information) 4. [Get Ping-Pong Memory Information](#ping-pong-memory) 5. [Get LLM Information](#get-llm-info) 6. [Get and Modify Current Hyperparameter](#get-and-modify-current-hyperparameter) 4. [Inference](#inference) 5. [Calbin (STILL UNDER DEVELOPMENT)](#calbin) 1. [Create Calbin](#create-calbin) 2. [Get Calbin Information](#get-calbin) 3. [Get Model Information](#get-model) 6. [Memory Management](#memory-management) 1. [Register Device Memory](#register-device-memory) 2. [Allocate Device Memory](#allocate-device-memory) 3. [Copy from Host Memory to Device Memory](#copy-from-host-memory-to-device-memory) 4. [Copy from Device Memory to Host Memory](#copy-from-device-memory-to-host-memory) 5. [Write Device Register](#write-device-register) 6. [Read Device Register](#read-device-register) 7. [Free Device Memory](#free-device-memory) 7. [Error Handling](#error-handling) 1. [CALRT_CHECK](#calrt-check) 2. [REPORT_CALRT_ERROR](#report-runtime-error) 3. [REPORT_CALRT_ERROR_IF](#report-runtime-error-if) [Back to Main](../RuntimeAPI.md) --- ## **Configuration** ## This section describes configure function of calculet runtime programming interface. ### ***configure*** ### > `CalrtError_e configure(CalculetDevice *vDev, Calbin *pParser)` configure calbin to device #### ***Parameter*** #### `vDev` -- calculet device pointer, including `VirtualDevice` and `CalrtDevice` #### ***Return*** #### [CalrtSuccess](./enum.md#calrterror_e), [CalrtErrorInvalidConfiguration](./enum.md#calrterror_e) #### ***Description*** #### return `CalrtSuccess` if configure successfully. [Back to Top](#table-of-modules) --- ## **Device** ## This section describes device control functions of calculet runtime under [CalrtDevice](./class.md#calrtdevice) object. ### ***Scan Device*** ### > `static std::optional> scanDevice()` scan valid calculet device #### ***Return*** #### [CalrtDeviceId_s](./struct.md#calrtdeviceid_s) #### ***Description*** #### if success, a list of Calculet device ids would be returned. It does matter to construct runtime device objects to control physical device. [Back to Top](#table-of-modules) --- ### ***Create Device*** ### > `static std::unique_ptr CreateDevice(const CalrtDeviceId_s &id)` create runtime device object #### ***Parameter*** #### `id` -- a calculet device id. #### ***Return*** #### [`std::unique_ptr`](./class.md#calrtdevice) a valid runtime device unique pointer for controlling physical device #### ***Description*** #### Argument `id` must be obtained by `scanDevice()` which guarentee a number of valid calculet device ids are captured. You may choose to create your own device id by initialize `struct CalrtDeviceId_s`. But it may be failed to construct runtime device object. [Back to Top](#table-of-modules) --- ### ***Trigger Device Execution*** ### > `void StartJob(uint64_t jobAddr, uint64_t jobSize) const` trigger device execution #### ***Parameter*** #### `jobAddr` -- program start memory address in device. `jobSize` -- program byte size. #### ***Description*** #### Let device to execute a program at specific memory address. [Back to Top](#table-of-modules) --- ### ***Wait Device Execution Finish*** ### > `void WaitJobDone(std::atomic_bool& shutdown) const` `void WaitJobDone(std::atomic_bool& shutdown, uint64_t& jobEndInfo) const` wait untill device execution finished and get data back to output buffer. #### ***Parameter*** #### `shutdown` -- indicate if core-engine is shutdown. `jobEndInfo` -- a finished job id. #### ***Description*** #### A blocking function used to get data back to host memory when device finishes corresponding job. [Back to Top](#table-of-modules) --- ### ***Reset Device*** ### > `void Reset()` reset device, including memory allocation and firmware to factory setting. [Back to Top](#table-of-modules) --- ### ***Update Firmware*** ### > `void UpdateFirmware()` **STILL UNDER DEVELOPMENT** update device firmware. [Back to Top](#table-of-modules) --- ### ***Get Firmware Information*** ### > `CalrtError_e GetFirmwareInfo()` **STILL UNDER DEVELOPMENT** [Back to Top](#table-of-modules) --- ### ***Get Device Information*** ### > `const char *GetDevInfo()` Get Device Information, including memory usage, core frequency, etc. [Back to Top](#table-of-modules) --- ### ***Get Device Execution Status*** ### > `CalrtError_e GetDeviceStatus()` Get Device Statues #### ***Return*** #### [CalrtSuccess](./enum.md#calrterror_e), [CalrtErrorDeviceBusy](./enum.md#calrterror_e) [Back to Top](#table-of-modules) --- ## **Buffer** ## This section describes buffer function of calculet runtime programming object. It is used for inference, carrying valid data. ### ***Create Buffer*** ### > `std::unique_ptr createInputBuf(const CalbinModel &model)` `std::unique_ptr createOutputBuf(const CalbinModel &model)` create input or output buffer #### ***Parameter*** #### `model` -- [CalbinModel](./struct.md#calbinmodel) obtained by parsing calbin file #### ***Return*** #### [`std::unique_ptr`](./class.md#calrtinputbuf) runtime input buffer object pointer [`std::unique_ptr`](./class.md#calrtoutputbuf) runtime output buffer object pointer [Back to Top](#table-of-modules) --- ### ***Get Buffer Information*** ### > `const CalrtBufferInfo_s& GetBufferInfo() const` Get i/o buffer information #### ***Return*** #### [CalrtBufferInfo_s](./struct.md#calrtbufferinfo_s) [Back to Top](#table-of-modules) --- ### ***Tensors Information*** ### > `std::vector& GetTensors()` get i/o buffer tensor reference information and allow to modify it. #### ***Return*** #### [CalrtTensor](./class.md#calrttensor) #### ***Description*** #### With `CalrtTensor.GetDataPtr()`, it return a `char *` for input data or get output data. [Back to Top](#table-of-modules) --- ### ***Ping-Pong Memory*** ### > `std::vector& GetTensorMemInfo()` get i/o buffer ping-pong address #### ***Return*** #### [CalrtDevBuf_s](./struct.md#calrtdevbuf_s) [Back to Top](#table-of-modules) --- ### ***Get LLM info*** ### > `const CalbinLLM_s &GetLLM()` Get max batch size and max sequence length #### ***Return*** #### [CalbinLLM_s](./struct.md#calbinllm_s) #### ***Note*** #### Only input buffer has this member function. It is NULL if running CNN model. [Back to Top](#table-of-modules) --- ### ***Get and Modify Current Hyperparameter*** ### > `ModelHyperParameters_s &GetCurHyperParam()` Get and set current LLM needed hyperparameter #### ***Return*** #### [ModelHyperParameters_s](./struct.md#modelhyperparameters_s) #### ***Note*** #### Only input buffer has this member function. It is NULL if running CNN model. [Back to Top](#table-of-modules) --- ## **Inference** ## This section describes inference function of calculet runtime programming interface. > `void infer(std::unique_ptr &vDev, CalbinModel *model, CalrtInputBuf* inputBuffer, CalrtOutputBuf* outputBuffer);` ***Parameter*** `vDev` -- [VirtualDevice](./class.md#virtualdevice) virtual device manages one or multiple physical device. `model` -- [CalbinModel](./struct.md#calbinmodel) try to infer specific model `inputBuffer` -- [CalrtInputBuf](./class.md#calrtinputbuf) runtime input buffer pointer `outputBuffer` -- [CalrtOutputBuf](./class.md#calrtoutputbuf) runtime output buffer pointer ***Description*** Non-blocking function used to infer specific model with selected device or multiple devices. use `outputBuffer` to get inference result. [Back to Top](#table-of-modules) --- ## **Calbin** ## This section describes calbin class part of calculet runtime programming. ### ***Create Calbin*** ### > `static std::unique_ptr CreateParser(const std::string &path)` Construct an unique pointer of Calbin object #### ***Parameter*** ### `path` -- location of *calbin* file [Back to Top](#table-of-modules) --- ### ***Get Calbin*** ### > `CalrtCalbin& GetCalbin()` Get a reference of [CalrtCalbin](./struct.md#calrtcalbin) data struct [Back to Top](#table-of-modules) --- ### ***Get model*** ### > `std::optional GetModelByName(const char *modelName)` `std::vector GetAllModels()` Both member functions can provide a copy of the current model descripted in calbin file. Difference is one for the specific one and the other one is for all [CalbinModel](./struct.md#calbinmodel) data structure. [Back to Top](#table-of-modules) --- ## **Memory Management** ## This section describes [CalrtDevice](./class.md#calrtdevice) member functions of memory management. ### ***Register Device Memory*** ### > `CalrtError_e RegisterDram(uint64_t startAddr, uint64_t size)` register(reserve) device DRAM > `CalrtError_e RegisterSyncUnit(uint64_t startAddr, uint64_t size)` register(reserve) sync unit > `CalrtError_e RegisterSramBuf(uint64_t startAddr, uint64_t size)` register(reserve) SRAM #### ***Parameter*** #### `startAddr` -- reserved memory start address `size` -- reserved byte size #### ***Return*** #### [CalrtSuccess](./enum.md#calrterror_e), [CalrtErrorMemoryAlreadyRegistered](./enum.md#calrterror_e) #### ***Usage*** #### ```cpp /*CalrtDevice pointer*/ device_ptr->RegisterDram(0x1000, 32); ``` [Back to Top](#table-of-modules) --- ### ***Allocate Device Memory*** ### > `uint64_t CreateBuf(uint64_t size, uint32_t alignLog2Byte=0)` create DRAM buffer > `uint64_t CreateSramBuf(uint64_t size, uint32_t alignLog2Byte=0)` create SRAM buffer > `uint64_t ApplySyncUnit()` apply a sync unit #### ***Parameter*** #### `size` -- bytes size. `alignLog2Byte` -- align to specific byte size by 2^n. E.G. 64 byte align. then n is 6. #### ***Return*** #### `uint64_t` buffer start address. #### ***Note*** #### if fail to allocate memory, then program would be abort. #### ***Usage*** #### ```cpp /*CalrtDevice pointer*/ device_ptr->CreateBuf(32, 20) // allocate 32 byte memory aligned to 1MB on DRAM ``` [Back to Top](#table-of-modules) --- ### ***Copy from Host Memory to Device Memory*** ### > `void WriteToDevice(void *srcAddr, const uint64_t devAddr, const uint64_t size)` coyp host memory to device memory #### ***Parameter*** #### `srcAddr` -- host side buffer start address. `devAddr` -- device side memory start address. `size` -- buffer byte size. [Back to Top](#table-of-modules) --- ### ***Copy from Device Memory to Host Memory*** ### > `void ReadFromDevice(void *srcAddr, const uint64_t devAddr, const uint64_t size)` #### ***Parameter*** #### `srcAddr` -- host side buffer start address. `devAddr` -- device side memory start address. `size` -- buffer byte size. ### ***Write Device Register*** ### > `void WriteReg(const uint64_t regAddr, uint32_t data)` write data to register by address #### ***Parameter*** #### `regAddr` -- register address. `data` -- input data [Back to Top](#table-of-modules) --- ### ***Read Device Register*** ### > `void ReadReg(const uint64_t regAddr, uint32_t &data)` read data from register by address #### ***Parameter*** #### `regAddr` -- register address. `data` -- output data [Back to Top](#table-of-modules) --- ### ***Free Device Memory*** ### > `void FreeBuf(uint64_t addr)` Free DRAM by device address > `void FreeSramBuf(uint64_t addr)` Free SRAM by device address > `void FreeSyncUnitByAddress(uint64_t addr)` Free sync unity by device address > `void FreeSyncUnitByIndex(uint32_t idx)` Free sync unity by index [Back to Top](#table-of-modules) --- ## **Error Handling** ## ### ***Calrt Check*** ### check error code if is CalrtSuccess. Otherwise abort program with optional error message. > `CALRT_CHECK(CalrtError_e, msg)` ### ***Report Runtime Error*** ### Report error code with optional error message and abort program. > `REPORT_CALRT_ERROR(CalrtError_e, msg)` ### ***Report Runtime Error if*** ### Report error code with optional error message and abort program if condition is false. > `REPORT_CALRT_ERROR_IF(CalrtError_e, condition, msg)` #### ***Description*** #### work like assert, only report error when condition is false. [Back to Top](#table-of-modules) ---