* Add centering of settings sidebar link when isActive
* abstract redundant logic into a reusable hook
* add jsdocs | refactor hook to consume behavior and block args
* remove unused import
* dev
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Extract Model Table to component
Add provider icons to header rows and installed models
Light mode supported
Mapping for model name id hints to provider
Update DMR to filter chat models by ability since not available via hub API
* linting + dev
* fix incorrect import
* Adjust fix path to use ESM import
* normalize fix-path imports and usage across the app
* extract path fix logic to utils for server and collector
* add helpers
* repin strip-ansi in collector
* fix log for localWhisper
lint
* Improve DMR support
- Autodetect models installed
- Grab all models from hub.docker to show available
- UI to handle render,search, install, and management of models
- Support functionality for chat, stream, and agentic calls
* forgot files
* fix loader circle being too large
fix tooltip width command
adjust location of docker installer open for web platform
* adjust imports
* remove create workspace step for onboarding
* remove unused image
* workspace creation into dedicated useEffect + use translated workspace name
* dev tag
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Migrate Astra to class (#4722)
migrate astra to class
* Migrate LanceDB to class (#4721)
migrate lancedb to class
* Migrate Pinecone to class (#4726)
migrate pinecone to class
* Migrate Zilliz to class (#4729)
migrate zilliz to class
* Migrate Weaviate to class (#4728)
migrate weaviate to class
* Migrate Qdrant to class (#4727)
migrate qdrant to class
* Migrate Milvus to class (#4725)
migrate milvus to class
* Migrate Chroma to class (#4723)
migrate chroma to class
* Migrate Chroma Cloud to class (#4724)
* migrate chroma to class
* migrate chroma cloud to class
* move limits to class field
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Migrate PGVector to class (#4730)
* migrate pgvector to class
* patch pgvector test
* convert connectionString, tableName, and validateConnection to static methods
* move instance properties to class fields
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Refactor Zilliz Cloud vector DB provider (#4749)
simplify zilliz implementation by using milvus as base class
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* VectorDatabase base class (#4738)
create generic VectorDatabase base class
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Extend VectorDatabase base class to all providers (#4755)
extend VectorDatabase base class to all providers
* patch lancedb import
* breakout name and add generic logger
* dev tag build
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* refactor localWhisper to use new custom FFMPEGWrapper class
* stub tests in github actions
* add back wavefile conversion to 16khz 32f to fix docker builds
* use afterEach for cleanup in ffmpeg tests
* remove unused FFMPEG_PATH env check
* use spawnSync for ffmpeg to capture and log output
* lint
* revert removal of try/catch around validateAudioFile for more helpful error msgs
* use readFileSync instead of createReadStream for less overhead
* change import to require for fix-path and stub import in tests
* refactor to singleton to preserve ffmpeg path
dev build
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Add local ESLint configuration and disable rules to allow for errorless state
* Remove unnecessary ESLint disable comments in AuthContext and usePromptInputStorage for cleaner code.
* Update eslint-plugin-react-hooks
* Configure prettier to work with eslint
* Removed trailing commas from eslint config
* Prettier to source code
* add a v2 lint script
* put back eslint-disable comments
* fix eslinter and prettier application
always apply --fix since we --write prettier, otherwise it fails
* precaution dev build
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* add refresh user functionality
* prettier
* add eslint disable comment for exhaustive-deps warning in AuthContext to stop nagging about navigate func
* remove unused imports and fix typo
* handle unsafe parse of undefined for in-session user deleted
* Refactor refreshUser function to handle errors and return structured response. Update AuthProvider to manage user data based on success status.
* Remove console error logging from promise catch in System model for cleaner error handling.
* change status from 404 to 400 and valid to success
* Refactor error handling in AuthProvider's refreshUser logic to remove redundant catch block and streamline user session management on failure.
* prettier
* reorder clauses - return errors
* refactor
account for all user modes
dev build
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* test base image bump
* patch frontend compilation due to Vite QEMU exception
* migrate all runners to arm64 to avoid QEMU slowdown
* add back layer cleanup
* remove layer cache fix
* Add Default System Prompt Management
- Introduced a new route for fetching and updating the default system prompt in the backend.
- Added a new Admin page for managing the default system prompt, including a form for editing and saving changes.
- Updated the SettingsSidebar to include a link to the new Default System Prompt page.
- Implemented fetching of available system prompt variables for use in the prompt editor.
- Enhanced the ChatSettings and ChatPromptSettings components to support the new default system prompt functionality.
This commit lays the groundwork for improved management of system prompts across workspaces.
* Remove validation for system prompt in ChatSettings component
* Add comment for system prompt in workspaces model
* linting, simplify logic for default assumption
* dev build
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* feat: add web app manifest and mobile PWA meta tags
* feat: serve dynamic manifest.json with custom branding for pwa
* feat: add ios status bar theming for pwa
* fix: prevent overscroll behavior for mobile
* fix: prevent ios safari auto-zoom on chat input
* fix: remove theme-color meta tags conflicting with ios status bar
* fix: add missing apple-mobile-web-app-capable meta tag for ios pwa
* fix: move catch-all route after manifest endpoint to prevent interception
* feat: add pwa detection helper and conditional styling for standalone mode
* PWA refactor
* undo changes to native CSS
* class fix
* proper response obj
* fix patch for import
* fix manifest errors
---------
Co-authored-by: Christian De Santis <christian.constantino98@gmail.com>
* Migrate gemini agents away from `Untooled`
* disable agents for gemma models as they are not supported for tool calling
* Dev build
resolve#4452 via function name prefix and then stripping within provider
* WIP agentic tool call streaming
- OpenAI
- Anthropic
- Azure OpenAI
* WIP rest of providers EXCLUDES Bedrock and GenericOpenAI
* patch untooled complete/streaming to use chatCallback provider from provider class and not assume OpenAI client struct
example: Ollama
* modify ollama to function with its own overrides
normalize completion/stream outputs across providers/untooled
* dev build
* fix message sanization for anthropic agent streaming
* wip fix anthropic agentic streaming sanitization
* patch gemini, webgenui, generic aibitat providers + disable providers unable to test
* refactor anthropic aibitat provider for empty message and tool call formatting
* Add frontend missing prop check
update Azure for streaming support
update Gemini to streamting support on gemini-* models
generic OpenAI disable streaming
verify localAI support
verify NVIDIA Nim support
* DPAIS, remove temp from call, support streaming'
* remove 0 temp to remove possibility of bad temp error/500s/400s
* Patch condition where model is non-streamable and no tools are present or called resulting in the provider `handleFunctionCallChat` being called - which returns a string.
This would then fail in Untooled.complete since response would be a string and not the expected `response.choices?.[0]?.message`
Modified this line to handle both conditions for stream/non-streaming and tool presence or lack thereof
* Allow generic Openai to be streamable since using untooled it should work fine
honor disabled streaming for provider where that concern may apply for regular chats
* rename function and more gemini-specific function to gemini provider
* add comments for readability
.complete on azure should be non-streaming as this is the sync response
* migrate CometAPI, but disable as we cannot test
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* Create parse endpoint in collector (#4212)
* create parse endpoint in collector
* revert cleanup temp util call
* lint
* remove unused cleanupTempDocuments function
* revert slug change
minor change for destinations
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Add parsed files table and parse server endpoints (#4222)
* add workspace_parsed_files table + parse endpoints/models
* remove dev api parse endpoint
* remove unneeded imports
* iterate over all files + remove unneeded update function + update telemetry debounce
* Upload UI/UX context window check + frontend alert (#4230)
* prompt user to embed if exceeds prompt window + handle embed + handle cancel
* add tokenCountEstimate to workspace_parsed_files + optimizations
* use util for path locations + use safeJsonParse
* add modal for user decision on overflow of context window
* lint
* dynamic fetching of provider/model combo + inject parsed documents
* remove unneeded comments
* popup ui for attaching/removing files + warning to embed + wip fetching states on update
* remove prop drilling, fetch files/limits directly in attach files popup
* rework ux of FE + BE optimizations
* fix ux of FE + BE optimizations
* Implement bidirectional sync for parsed file states
linting
small changes and comments
* move parse support to another endpoint file
simplify calls and loading of records
* button borders
* enable default users to upload parsed files but NOT embed
* delete cascade on user/workspace/thread deletion to remove parsedFileRecord
* enable bgworker with "always" jobs and optional document sync jobs
orphan document job: Will find any broken reference files to prevent overpollution of the storage folder. This will run 10s after boot and every 12hr after
* change run timeout for orphan job to 1m to allow settling before spawning a worker
* linting and cleanup pr
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* dev build
* fix tooltip hiding during embedding overflow files
* prevent crash log from ERRNO on parse files
* unused import
* update docs link
* Migrate parsed-files to GET endpoint
patch logic for grabbing models names from utils
better handling for undetermined context windows (null instead of Pos_INIFI)
UI placeholder for null context windows
* patch URL
---------
Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
* WIP on mobile connections
todo: register devices
todo: data sync or connection
* improve connection flow and registration
add streaming from service
TODO: user scoping
* dev build mobile support
* fix path
* handle relative URLs
* handle localhost access in product
* add device de-register
* sync styles
* move UI to be out of the normal path since beta only
* Add user scoping to mobile connection requests
Remigrate DB for user associations
Implement temp token registration to prevent unauthorized device registration requests
cleanup middlewares
* WIP on embedder selection
TODO: apply splitting and query prefixes (if applicable)
* wip on upsert
* Support base model
support nomic-text-embed-v1
support multilingual-e5-small
Add prefixing for both embedding and query for RAG tasks
Add chunking prefix to all vector dbs to apply prefix when possible
Show dropdown and auto-pull on new selection
* norm translations
* move supported models to constants
handle null seelction or invalid selection on dropdown
update comments
* dev
* patch text splitter maximums for now
* normalize translations
* add tests for splitter functionality
* normalize
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* Enable UI/UX for model swapping in chat window
* forgot component
* patch useGetProviders hook to set loading on change of provider
* dev build
* normalize translations
* patch how model default is provided
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* implement importing of agent flows from community hub
* auto enable flow on import
* remove unused blocks for docker
prevent importing or saving of agent flows that have unsupported blocks for version or platform
* dev build
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* wip: create direct output switch on last block and send response to ui
* lint
* Return flow on direct output enabled
prevent new blocks below direct output block
Update executor/aibitat to handle skipping of handler outputs
* dev build
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Add context window finder from litellm maintained list
apply to all cloud providers, have client cache for 3 days
* docker container bootup warning
* update invalid ENV warning
* rebased with current HEAD
* update newline printing
* PGVector support for vector db storage
* forgot files
* comments
* dev build
* Add ENV connection and table schema validations for vector table
add .reset call to drop embedding table when changing the AnythingLLM embedder
update instrutions
Add preCheck error reporting in UpdateENV
add timeout to pg connection
* update setup
* update README
* update doc
* auto STT submit control
* simplified STT auto and TTS autoplay
* add chatId to piper elements
* change defaults
* add chat customization page for enabling/disabling new features + options on account modal
* add translation placeholders
* show keybind command
* dev build
* Account modal ui prevent page reload + show Tooltip
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* Translate main page landing
* normalize all languages
* Add a few translations, zh,tw,ja,es,de
* fix issues where translation lang change would not effect constants until full reload instead of on update
* dev
* fix noWorkspaceError key in translation usage
* Enable bypass of ip limitations via ENV in collector startup
resolves#3625
connect #3626
* dev build
* bump dockerx build action
* enable runtime setting config of collector requests
* comments and linting for option passing
* unset
* unset
* update docs link
* linting and docs
* wip getting started checklist page
* lint
* add new ui skeleton for new home page/checklist
* make legacy home page appearance setting
* dynamic checklist rendering/close checklist
* make home page functional + update cta buttons in ws settings
* lint
* remove unneeded routes
* wip fixing checklist items (create ws and embed doc broken) + news section updates
* lint
* FINALLY fix all functionality & remove hook to simplify logic
* lint
* hide/show options based on user role/mum enabled
* add hover states and redo checklist ui
* remove welcome checklist page
* add validation to task completion
* polish behavior of checklist/fix roles on checklist items
* lint
* light mode/use tailwind color classes
* remove tutorials link
* Modify how legacy page works
* small UI updates
* remove unused paths
cleanup explore features
* revert save button changes
* conditionally render legacy default chat page when in multiuser
* remove role checks in checklist
* remove role checks in quick links
* remove unused hook
* dark mode new home page checklist ui updates
* Add news logging to repo for record keeping
update new module to pull from CDN
* simplify landing markup
* light mode styles
* remove border in light mode from merge conflict
* Update ignores
* slide up dismiss
* prevent checklist popin
* confetti?
* fix url hash on navigate
* watch for event changes for updating checklist
* useMemo and callback memory optimization
* move handlers to constants via params fwd
* dev
* update github text
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* WIP MCP full compatibility layer
* implement MCP agent function wrapping and invocation methods
* Add `uvx` to docker bin for MCP executions
* dev build
* prune removed data
* Wrap MCP servers to lazy load items to not block the UI
Mobile bug fixes
* arm64 test build
* reset dev builder
* remove unused prop
