Extensions
pg_tiktoken

pg_tiktoken

pg_tiktoken : tiktoken tokenizer for use with OpenAI models in postgres

Overview

ID Extension Package Version Category License Language
1870
pg_tiktoken
pg_tiktoken
 0.0.1
RAG
Apache-2.0
Rust
Attribute Has Binary Has Library Need Load Has DDL Relocatable Trusted
--s-d--
No
Yes
No
Yes
no
no
Relationships
See Also
vectorize
pg_summarize
pg4ml
pgml
vector
vchord
vectorscale
pg_graphql

Packages

Type Repo Version PG Major Compatibility Package Pattern Dependencies
EXT
PIGSTY
 0.0.1
18
17
16
15
14
 pg_tiktoken -
RPM
PIGSTY
 0.0.1
18
17
16
15
14
 pg_tiktoken_$v -
DEB
PIGSTY
 0.0.1
18
17
16
15
14
 postgresql-$v-pg-tiktoken -
Linux / PG PG18 PG17 PG16 PG15 PG14
el8.x86_64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
el8.aarch64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
el9.x86_64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
el9.aarch64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
el10.x86_64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
el10.aarch64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
d12.x86_64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
d12.aarch64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
d13.x86_64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
d13.aarch64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
u22.x86_64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
u22.aarch64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
u24.x86_64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
u24.aarch64
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
PIGSTY 0.0.1
Package Version OS ORG SIZE File URL
pg_tiktoken_18 0.0.1 el8.x86_64 pigsty 1.7 MiB pg_tiktoken_18-0.0.1-2PIGSTY.el8.x86_64.rpm
pg_tiktoken_18 0.0.1 el8.aarch64 pigsty 1.6 MiB pg_tiktoken_18-0.0.1-2PIGSTY.el8.aarch64.rpm
pg_tiktoken_18 0.0.1 el9.x86_64 pigsty 1.6 MiB pg_tiktoken_18-0.0.1-2PIGSTY.el9.x86_64.rpm
pg_tiktoken_18 0.0.1 el9.aarch64 pigsty 1.5 MiB pg_tiktoken_18-0.0.1-2PIGSTY.el9.aarch64.rpm
pg_tiktoken_18 0.0.1 el10.x86_64 pigsty 1.7 MiB pg_tiktoken_18-0.0.1-2PIGSTY.el10.x86_64.rpm
pg_tiktoken_18 0.0.1 el10.aarch64 pigsty 1.5 MiB pg_tiktoken_18-0.0.1-2PIGSTY.el10.aarch64.rpm
postgresql-18-pg-tiktoken 0.0.1 d12.x86_64 pigsty 1.4 MiB postgresql-18-pg-tiktoken_0.0.1-1PIGSTY~bookworm_amd64.deb
postgresql-18-pg-tiktoken 0.0.1 d12.aarch64 pigsty 1.2 MiB postgresql-18-pg-tiktoken_0.0.1-1PIGSTY~bookworm_arm64.deb
postgresql-18-pg-tiktoken 0.0.1 d13.x86_64 pigsty 1.4 MiB postgresql-18-pg-tiktoken_0.0.1-1PIGSTY~trixie_amd64.deb
postgresql-18-pg-tiktoken 0.0.1 d13.aarch64 pigsty 1.2 MiB postgresql-18-pg-tiktoken_0.0.1-1PIGSTY~trixie_arm64.deb
postgresql-18-pg-tiktoken 0.0.1 u22.x86_64 pigsty 1.6 MiB postgresql-18-pg-tiktoken_0.0.1-1PIGSTY~jammy_amd64.deb
postgresql-18-pg-tiktoken 0.0.1 u22.aarch64 pigsty 1.4 MiB postgresql-18-pg-tiktoken_0.0.1-1PIGSTY~jammy_arm64.deb
postgresql-18-pg-tiktoken 0.0.1 u24.x86_64 pigsty 1.6 MiB postgresql-18-pg-tiktoken_0.0.1-1PIGSTY~noble_amd64.deb
postgresql-18-pg-tiktoken 0.0.1 u24.aarch64 pigsty 1.4 MiB postgresql-18-pg-tiktoken_0.0.1-1PIGSTY~noble_arm64.deb
Package Version OS ORG SIZE File URL
pg_tiktoken_17 0.0.1 el8.x86_64 pigsty 1.7 MiB pg_tiktoken_17-0.0.1-2PIGSTY.el8.x86_64.rpm
pg_tiktoken_17 0.0.1 el8.aarch64 pigsty 1.6 MiB pg_tiktoken_17-0.0.1-2PIGSTY.el8.aarch64.rpm
pg_tiktoken_17 0.0.1 el9.x86_64 pigsty 1.6 MiB pg_tiktoken_17-0.0.1-2PIGSTY.el9.x86_64.rpm
pg_tiktoken_17 0.0.1 el9.aarch64 pigsty 1.5 MiB pg_tiktoken_17-0.0.1-2PIGSTY.el9.aarch64.rpm
pg_tiktoken_17 0.0.1 el10.x86_64 pigsty 1.7 MiB pg_tiktoken_17-0.0.1-2PIGSTY.el10.x86_64.rpm
pg_tiktoken_17 0.0.1 el10.aarch64 pigsty 1.5 MiB pg_tiktoken_17-0.0.1-2PIGSTY.el10.aarch64.rpm
postgresql-17-pg-tiktoken 0.0.1 d12.x86_64 pigsty 1.4 MiB postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~bookworm_amd64.deb
postgresql-17-pg-tiktoken 0.0.1 d12.aarch64 pigsty 1.2 MiB postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~bookworm_arm64.deb
postgresql-17-pg-tiktoken 0.0.1 d13.x86_64 pigsty 1.4 MiB postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~trixie_amd64.deb
postgresql-17-pg-tiktoken 0.0.1 d13.aarch64 pigsty 1.2 MiB postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~trixie_arm64.deb
postgresql-17-pg-tiktoken 0.0.1 u22.x86_64 pigsty 1.6 MiB postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~jammy_amd64.deb
postgresql-17-pg-tiktoken 0.0.1 u22.aarch64 pigsty 1.4 MiB postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~jammy_arm64.deb
postgresql-17-pg-tiktoken 0.0.1 u24.x86_64 pigsty 1.6 MiB postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~noble_amd64.deb
postgresql-17-pg-tiktoken 0.0.1 u24.aarch64 pigsty 1.4 MiB postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~noble_arm64.deb
Package Version OS ORG SIZE File URL
pg_tiktoken_16 0.0.1 el8.x86_64 pigsty 1.7 MiB pg_tiktoken_16-0.0.1-2PIGSTY.el8.x86_64.rpm
pg_tiktoken_16 0.0.1 el8.aarch64 pigsty 1.6 MiB pg_tiktoken_16-0.0.1-2PIGSTY.el8.aarch64.rpm
pg_tiktoken_16 0.0.1 el9.x86_64 pigsty 1.6 MiB pg_tiktoken_16-0.0.1-2PIGSTY.el9.x86_64.rpm
pg_tiktoken_16 0.0.1 el9.aarch64 pigsty 1.5 MiB pg_tiktoken_16-0.0.1-2PIGSTY.el9.aarch64.rpm
pg_tiktoken_16 0.0.1 el10.x86_64 pigsty 1.7 MiB pg_tiktoken_16-0.0.1-2PIGSTY.el10.x86_64.rpm
pg_tiktoken_16 0.0.1 el10.aarch64 pigsty 1.5 MiB pg_tiktoken_16-0.0.1-2PIGSTY.el10.aarch64.rpm
postgresql-16-pg-tiktoken 0.0.1 d12.x86_64 pigsty 1.4 MiB postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~bookworm_amd64.deb
postgresql-16-pg-tiktoken 0.0.1 d12.aarch64 pigsty 1.2 MiB postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~bookworm_arm64.deb
postgresql-16-pg-tiktoken 0.0.1 d13.x86_64 pigsty 1.4 MiB postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~trixie_amd64.deb
postgresql-16-pg-tiktoken 0.0.1 d13.aarch64 pigsty 1.2 MiB postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~trixie_arm64.deb
postgresql-16-pg-tiktoken 0.0.1 u22.x86_64 pigsty 1.6 MiB postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~jammy_amd64.deb
postgresql-16-pg-tiktoken 0.0.1 u22.aarch64 pigsty 1.4 MiB postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~jammy_arm64.deb
postgresql-16-pg-tiktoken 0.0.1 u24.x86_64 pigsty 1.6 MiB postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~noble_amd64.deb
postgresql-16-pg-tiktoken 0.0.1 u24.aarch64 pigsty 1.4 MiB postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~noble_arm64.deb
Package Version OS ORG SIZE File URL
pg_tiktoken_15 0.0.1 el8.x86_64 pigsty 1.7 MiB pg_tiktoken_15-0.0.1-2PIGSTY.el8.x86_64.rpm
pg_tiktoken_15 0.0.1 el8.aarch64 pigsty 1.6 MiB pg_tiktoken_15-0.0.1-2PIGSTY.el8.aarch64.rpm
pg_tiktoken_15 0.0.1 el9.x86_64 pigsty 1.6 MiB pg_tiktoken_15-0.0.1-2PIGSTY.el9.x86_64.rpm
pg_tiktoken_15 0.0.1 el9.aarch64 pigsty 1.5 MiB pg_tiktoken_15-0.0.1-2PIGSTY.el9.aarch64.rpm
pg_tiktoken_15 0.0.1 el10.x86_64 pigsty 1.7 MiB pg_tiktoken_15-0.0.1-2PIGSTY.el10.x86_64.rpm
pg_tiktoken_15 0.0.1 el10.aarch64 pigsty 1.5 MiB pg_tiktoken_15-0.0.1-2PIGSTY.el10.aarch64.rpm
postgresql-15-pg-tiktoken 0.0.1 d12.x86_64 pigsty 1.4 MiB postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~bookworm_amd64.deb
postgresql-15-pg-tiktoken 0.0.1 d12.aarch64 pigsty 1.2 MiB postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~bookworm_arm64.deb
postgresql-15-pg-tiktoken 0.0.1 d13.x86_64 pigsty 1.4 MiB postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~trixie_amd64.deb
postgresql-15-pg-tiktoken 0.0.1 d13.aarch64 pigsty 1.2 MiB postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~trixie_arm64.deb
postgresql-15-pg-tiktoken 0.0.1 u22.x86_64 pigsty 1.6 MiB postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~jammy_amd64.deb
postgresql-15-pg-tiktoken 0.0.1 u22.aarch64 pigsty 1.4 MiB postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~jammy_arm64.deb
postgresql-15-pg-tiktoken 0.0.1 u24.x86_64 pigsty 1.6 MiB postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~noble_amd64.deb
postgresql-15-pg-tiktoken 0.0.1 u24.aarch64 pigsty 1.4 MiB postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~noble_arm64.deb
Package Version OS ORG SIZE File URL
pg_tiktoken_14 0.0.1 el8.x86_64 pigsty 1.7 MiB pg_tiktoken_14-0.0.1-2PIGSTY.el8.x86_64.rpm
pg_tiktoken_14 0.0.1 el8.aarch64 pigsty 1.6 MiB pg_tiktoken_14-0.0.1-2PIGSTY.el8.aarch64.rpm
pg_tiktoken_14 0.0.1 el9.x86_64 pigsty 1.6 MiB pg_tiktoken_14-0.0.1-2PIGSTY.el9.x86_64.rpm
pg_tiktoken_14 0.0.1 el9.aarch64 pigsty 1.5 MiB pg_tiktoken_14-0.0.1-2PIGSTY.el9.aarch64.rpm
pg_tiktoken_14 0.0.1 el10.x86_64 pigsty 1.7 MiB pg_tiktoken_14-0.0.1-2PIGSTY.el10.x86_64.rpm
pg_tiktoken_14 0.0.1 el10.aarch64 pigsty 1.5 MiB pg_tiktoken_14-0.0.1-2PIGSTY.el10.aarch64.rpm
postgresql-14-pg-tiktoken 0.0.1 d12.x86_64 pigsty 1.4 MiB postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~bookworm_amd64.deb
postgresql-14-pg-tiktoken 0.0.1 d12.aarch64 pigsty 1.2 MiB postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~bookworm_arm64.deb
postgresql-14-pg-tiktoken 0.0.1 d13.x86_64 pigsty 1.4 MiB postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~trixie_amd64.deb
postgresql-14-pg-tiktoken 0.0.1 d13.aarch64 pigsty 1.2 MiB postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~trixie_arm64.deb
postgresql-14-pg-tiktoken 0.0.1 u22.x86_64 pigsty 1.6 MiB postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~jammy_amd64.deb
postgresql-14-pg-tiktoken 0.0.1 u22.aarch64 pigsty 1.4 MiB postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~jammy_arm64.deb
postgresql-14-pg-tiktoken 0.0.1 u24.x86_64 pigsty 1.6 MiB postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~noble_amd64.deb
postgresql-14-pg-tiktoken 0.0.1 u24.aarch64 pigsty 1.4 MiB postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~noble_arm64.deb

Source

Repository
github.com/kelvich/pg_tiktoken
 Source Tarball
pg_tiktoken-0.0.1.tar.gz
 
pig build pkg pg_tiktoken;		# build rpm/deb

Install

Make sure PGDG and PIGSTY repo available:

pig repo add pgsql -u   # add both repo and update cache

Install this extension with pig:

pig install pg_tiktoken;		# install via package name, for the active PG version

pig install pg_tiktoken -v 18;   # install for PG 18
pig install pg_tiktoken -v 17;   # install for PG 17
pig install pg_tiktoken -v 16;   # install for PG 16
pig install pg_tiktoken -v 15;   # install for PG 15
pig install pg_tiktoken -v 14;   # install for PG 14

Create this extension with:

CREATE EXTENSION pg_tiktoken;
Last updated on
pg_summarizepg4ml