Token Splitter Node

Der Token Splitter Node teilt eine rohe Textzeichenkette, indem er zuerst den Text in BPE-Token konvertiert, diese Token dann in Chunks aufteilt und die Token innerhalb eines einzelnen Chunks wieder in Text konvertiert.

Auf dieser Seite finden Sie die Node-Parameter für den Token Splitter Node sowie Links zu weiteren Ressourcen.

Parameter resolution in sub-nodes

Sub-nodes behave differently to other nodes when processing multiple items using an expression.

Most nodes, including root nodes, take any number of items as input, process these items, and output the results. You can use expressions to refer to input items, and the node resolves the expression for each item in turn. For example, given an input of five name values, the expression {{ $json.name }} resolves to each name in turn.

In sub-nodes, the expression always resolves to the first item. For example, given an input of five name values, the expression {{ $json.name }} always resolves to the first name.

Node-Parameter

• Chunk-Größe: Geben Sie die Anzahl der Zeichen in jedem Chunk ein.
• Chunk-Überlappung: Geben Sie an, wie viel Überlappung zwischen den Chunks bestehen soll.

Zugehörige Ressourcen

Weitere Informationen zum Dienst finden Sie in der Token-Dokumentation von LangChain und in der Textsplitter-Dokumentation von LangChain.

View Localmind Automate's Advanced AI documentation.

AI glossary

• completion: Completions are the responses generated by a model like GPT.
• hallucinations: Hallucination in AI is when an LLM (large language model) mistakenly perceives patterns or objects that don't exist.
• vector database: A vector database stores mathematical representations of information. Use with embeddings and retrievers to create a database that your AI can access when answering questions.
• vector store: A vector store, or vector database, stores mathematical representations of information. Use with embeddings and retrievers to create a database that your AI can access when answering questions.