4.9 KiB
Table of Structs
- CalrtCalbin
- CalbinModel
- CalbinSection_s
- CalbinLLM_s
- KV_Cache_s
- CalbinTensorInfo_s
- CalrtDevBuf_s
- CalrtBufferInfo_s
- TaskInfo_s
- CalrtDeviceId_s
- CalrtDeviceInfo_s
- ModelHyperParameters_s
CalrtCalbin
Descript a calbin file
Variables
std::string name
calbin file name
std::vector<CalbinModel> models
related models structs.
CalbinModel
Descript a model related information.
Variables
bool isConfigured
indicate if this modole is configured on device.
CalDevAccType_e devAccType
CalbinModelType_e modelType
CalbinModelArch_e modelArch
ModelChipMode_e chipMode
std::string modelName
indicate model name
CalbinLLM_s llm
indicate llm related information. Only invalid when CalbinModelType_e == CNN
std::unordered_multimap<CalbinSectionType_e, CalbinSection_s> sections
indicate this model component sections. CalbinSectionType_e, CalbinSection_s
std::vector<int32_t> tarChipN
indicate this model would be configured on which chips.
CalbinSection_s
Descript the content of specific model.
Variables
CalbinSectionPlace_e place
indicate where this section is located at. CalbinSectionPlace_e
CalbinSectionType_e type
std::string name
indicate section name
std::string filePath
indicate related binary or txt file path
int64_t offset
indicate .bss entry address
int64_t devAddr
indicate located device memory address
int64_t size
indicate section byte size
int64_t progEntry
indicate program header entry address
std::vector<uint32_t> chipMask
indicate which chip would hold the current section
std::vector<CalbinTensorInfo_s> tensorInfos
indicate a list of tensor info. CalbinTensorInfo_s
CalbinLLM_s
Descript max limits for batch size, sequence length and related KV cache information.
Variables
uint32_t max_batch_size
indicate max batch size the current model support
uint32_t max_seq_len
indicate max sequence length the current model support
KV_Cache_skv_cache`
KV_Cache_s
Descript kv cache memory information.
Variables
PrimitiveType dtype
indicate data type. default value is INVALID. PrimitiveType
uint32_t layer
indicate its transformer layer
int64_t kv_base[2]
indicate k cache base memory address and v cache base memory address
int64_t size
indicate kv cache byte size
std::vector<int32_t> shape
indicate kv cahce shape. Current is {head, batch, head_dim, d0, d2}
CalbinTensorInfo_s
Descript each tensor per model related informations.
Variable
std::string name
indicate tensor name
std::vector<int32_t> shape
indicate tensor shape
PrimitiveType dataType
indicate data type of the current tensor. PrimitiveType
int64_t ping; int64_t pong
indicate ping-pong memory address
int64_t size
indicate each ping-pong byte size
CalrtDevBuf_s
Descript Buffer info. Reserved.
CalrtBufferInfo_s
Used to construct calrt i/o buffer.
Variables
CalrtBufferDirection_e direction
indicate data flow direction. CalrtBufferDirection_e
std::string name
indicate buffer name
std::vector<CalbinTensorInfo_s> tensorInfos
indicate all tensors' info about the current model for either input or output.
`uint64_t devAddr0; uint64_t devAddr1; uint64_t size"
indicate each tensor's ping pong memory address and each memory block byte size. They are duplicated in CalbinTensorInfo_s.
TaskInfo_s
Descript an inference taks condition. For JobEgnine use.