Open

In Review

Planned

PRD in Progress

Dev In Progress

Completed

Main Roadmap

In order to make a workflow more comprehensive to meet different business needs, there will be a need to allow different paths based on the workflow variable or value

Conditional or Logic block to enable Workflow brances

Mei Wei Lim

Notification channels 1. Email 2. Platform notifications 3. In page notifications Features: 1. Allow user to opt in/out from email notifications 2. Allow user to read the notifications in Weave What notifications?<ol><li>Model is retired</li><li>New model is added</li><li>Credit balance running low</li><li>etc</li></ol>

Notification System for Users

We should include a list of templates that meets the general use cases of the users to start their prompting or workflow usage.

Starter pack templates for users' reference

maybe we can add a pending page thingy for automated fine tuning soon? As a placeholder icon right after prompt playground and workflow builder

Placeholder icon and page for fine tuning

An Anonymous User

Content: This feature is aimed at making teamwork easier on the platform by letting multiple team members work together on the same prompt and same workflow. This helps in organising teamwork and centralised communication. Under a new menu titled "Teams", I propose the following options:<ul><li>Create New Team<ul><li>This allows a user to start a new team, name the team, and add some members to start with.</li></ul></li><li>Invite New Users into Team<ul><li>This option lets team leaders or allowed members invite more people to join the team using their email.</li></ul></li><li>Remove Member from Team<ul><li>This will let authorized persons remove a member from the team and, making sure the team's work stays safe. </li></ul></li></ul>

Teams Function for collaborative feature

Suzy Lam

For security purposes, to avoid hackers to tamper the API keys and access to the endpoints

[Weave] Domain Whitelisting

Currently most LLM doesnt have internet connection, the trained data only until 2021, with this it is really useful for a better results with real time data.we can use API product like SerpAPI.

[Weave] Internet Connection for LLM

Johnson Lai

Unsure if this is already a pending feature, but having a default prompt template inserted when the user selects one of the llama2 models would be great. From a UX flow, the default prompt template should appear when one of the Llama models is selected, and disappear when an alternative model is selected.

[Weave] Default prompt template insertion in UX flow.

Feature: Prompt playground response time from LLM - would be great to know how much time a prompt spent to run with that specific LLM. Helps users to properly decide between models as well as structure prompt complexity<a target="_blank" rel="noopener noreferrer nofollow" class="text-indigo-500 dark:text-indigo-300 cursor-pointer" href="https://chasm-talk.slack.com/archives/C05PHU191GX/p1695637378933809">https://chasm-talk.slack.com/archives/C05PHU191GX/p1695637378933809</a> Discussion log ^

[Weave] Feature: Prompt playground response time from LLM

John Koh

Allow users to<ul><li>Save a workflow as a draft</li><li>Publish a version of the workflow</li><li>Copy a version as draft</li><li>Roll back to previous version of workflow</li></ul>This is for users to backtrack their changes

Workflow versioning

Description: Currently, the arrangement of AI models in the All models menu is scattered. Models from the same provider not grouped together. For example, Meta has models on the top on the list and also at the bottom of the list. This setup can lead to confusion and inefficiency when users are trying to locate specific models. Proposed Change: Reorganising the AI models by sequencing them according to their respective providers. This would entail placing all LLM models in a contiguous block. This will improve navigational efficiency and user experience by making it easier to find and compare models from the same provider. <img class="" src="https://fb-usercontent.fra1.cdn.digitaloceanspaces.com/1717749904766..png">

Reorganizing AI Models in 'All' view

<h2>Context:</h2>Currently there is no option of deleting or duplicating created prompts and workflows.<h2> Why is it beneficial?</h2>It would be useful for people who wanted to test slightly different versions of prompts / workflow by duplicating them quickly, or delete unwanted prompts / workflows.

Duplicate Prompts & Workflows

Devs are having hard time to understand how to properly use the API.Suggestions:<ul><li>Increase the visual hierarchy &amp; focus of endpoint URL, payload structure, response data structure.</li><li>Reduce the language options (currently we have multiple implementation for each language, but actually that doesn't help much) </li></ul><img src="https://fb-usercontent.fra1.cdn.digitaloceanspaces.com/1708914771839..png"><img src="https://fb-usercontent.fra1.cdn.digitaloceanspaces.com/1708914800306..png">

Improve API panel to help developer better understand on the usage

Yongsheng Lui

As a user, I want to be able to sort the prompts in the prompts list according to:<ul><li>last modified date</li><li>created date</li><li>last published date </li></ul><img src="https://fb-usercontent.fra1.cdn.digitaloceanspaces.com/1706759950586..png">

As a user, I want to be able to sort the prompts list in the workflow module

When a prompt has unpublished changes, show an indicator on the prompt node to inform users in the Workflow module

Show status for prompt node in workflow

<a target="_blank" rel="noopener noreferrer nofollow" class="text-indigo-500 dark:text-indigo-300 cursor-pointer" href="https://www.figma.com/file/6ENzA3PXIZEgUlJ0MxjGKm/Weave-(MVP-%E2%80%93-Q4-2023)?type=design&amp;node-id=640%3A2432&amp;mode=design&amp;t=WRmgxuBncJd9fdIR-1">Suggestion:</a> To have a sticky note for jotting down ideas, reminders or for better collaborations. How it looks like: <a target="_blank" rel="noopener noreferrer nofollow" class="text-indigo-500 dark:text-indigo-300 cursor-pointer" href="https://www.figma.com/file/6ENzA3PXIZEgUlJ0MxjGKm/Weave-(MVP-%E2%80%93-Q4-2023)?type=design&amp;node-id=640%3A2432&amp;mode=design&amp;t=WRmgxuBncJd9fdIR-1">https://www.figma.com/file/6ENzA3PXIZEgUlJ0MxjGKm/Weave-(MVP-%E2%80%93-Q4-2023)?type=design&amp;node-id=640%3A2432&amp;mode=design&amp;t=WRmgxuBncJd9fdIR-1</a> <img src="https://fb-usercontent.fra1.cdn.digitaloceanspaces.com/1701752190090.png">

Low Priority

Sticky note in the Workflow Module

Issue:<ul><li>After changing the values for the parameters, it is difficult to reset the value back to default as there is no obvious indicator.</li><li>Current implementation is to double click on the middle of the bar but it is too difficult to manoeuvre. There is also no tooltips to teach user to do this.</li></ul>Suggested implementation:<ul><li>Have a one-click reset all button at the top to reset all values to default.</li><li>At each line, have a small button at each of the parameter to reset only that particular value to default.</li></ul><img src="https://fb-usercontent.fra1.cdn.digitaloceanspaces.com/1706760158490..png">

Reset the parameter settings in one click

Allow user to delete prompt or workflow in Weave

High Priority

Delete Prompt or Workflow

As a user, I want to be able to filter prompts / workflow by statusAs a uesr, I want to be able to filter datasets by file type

Filter function for core modules

As a user, I want to be able to sort my prompts, workflow, datasets by:<ul><li>name (alphabetical order or reverse)</li><li>created date (earliest to latest, or reverse)</li><li>updated date (latest to earliest, or reverse)</li></ul>

Sorting function for core modules

By default, users can view the core modules in grid view. Users can have the option to toggle between grid view or list view. Core modules include: Prompt, Workflow and Datast

Option for users to view core modules in grid or list view

<img src="https://fb-usercontent.fra1.cdn.digitaloceanspaces.com/1708915142606..png"> As a user, I want to be able to search for a prompt by title As a user, I want to be able to search for a workflow by titleAs a user, I want to be able to search for a dataset by title

Search function for all core modules

Currently there is no way to pull the output from the first prompt in the workflow, and insert it into the next prompt.“How do I prompt in a way that the output of the first process will be used as the input of the second process?”This is a basic and super important feature that needs to be inserted into the workflow, so that we can start testing.

[Weave] Using the output of a prompt as a input for other prompts in the workflow

Dataset module will consists of datasets/files that can be used within prompts. Dataset can be in the form of txt, pdf and etc. <a target="_blank" rel="noopener noreferrer nofollow" class="text-indigo-500 dark:text-indigo-300 cursor-pointer" href="https://hooga-gaming.atlassian.net/wiki/spaces/Product/pages/155811853/Weave+-+PRD+-+RAG+Doc+Processing+and+Dataset+Module">https://hooga-gaming.atlassian.net/wiki/spaces/Product/pages/155811853/Weave+-+PRD+-+RAG+Doc+Processing+and+Dataset+Module</a>

Dataset & RAG Module

I think we should add gpt-4o since the price of it is cheaper than gpt-4 and gpt-4-turbo

Link: https://openai.com/api/pricing/

GPT-4o

When the actual output has emoji, Weave output doesn't support the emoji display. Refer to attachment Package link: <a target="_blank" rel="noopener noreferrer" class="text-indigo-500 dark:text-indigo-300 cursor-pointer c-link" href="https://uiwjs.github.io/react-json-view/">https://uiwjs.github.io/react-json-view/</a>Slack Thread: <a target="_blank" rel="noopener noreferrer nofollow" class="text-indigo-500 dark:text-indigo-300 cursor-pointer" href="https://chasm-talk.slack.com/archives/C05PHU191GX/p1701579288519599">https://chasm-talk.slack.com/archives/C05PHU191GX/p1701579288519599</a> <img src="https://fb-usercontent.fra1.cdn.digitaloceanspaces.com/1701653087948.png">

Allow Output to support Emoji

As per what Azad mentioned too above, having a health checkon the api (undefined: http://34.102.119.218:6507/health) , (undefined: http://34.102.119.218:6570/health) etc etc would be great.

Periodic Health Check on LLM's

Background:Our current product lacks the support for System Messages in ChatGPT/Llama2 and requires a custom instruction mechanism <code>[INST]&lt;&lt;SYS&gt;&gt;&lt;&lt;/SYS&gt;&gt;[/INST]</code> for Llama2. The absence of a structured messaging type framework restricts the flexibility and interaction model that can be crucial for developer-centric applications.Site like ChatGPT Playground <a target="_blank" rel="noopener noreferrer nofollow" class="text-indigo-500 dark:text-indigo-300 cursor-pointer" href="https://platform.openai.com/playground">https://platform.openai.com/playground</a><img src="https://fb-usercontent.fra1.cdn.digitaloceanspaces.com/1696057076216.png">Benefits:<ul><li>Improved interaction: This message types allow more interactive and intuitive communication model</li><li>Better UX: For Llama2 user, user dont need to type in <code>[INST]&lt;&lt;SYS&gt;&gt;</code> any more, also allow user to swap LLM easily</li><li>Future-proofing: ChatGPT playground already shown usecases</li></ul>Proposal:<ol><li>Support Multiple message and Message Type like System, User, Assistant for both ChatGPT and Llama2</li></ol><img src="https://fb-usercontent.fra1.cdn.digitaloceanspaces.com/1696057869881.png"><img src="https://fb-usercontent.fra1.cdn.digitaloceanspaces.com/1696057045912.png"><ol start="2"><li>Support extendable API for message type to allow conversational usecase, useful for things like ChatBot / Interview AI</li></ol><pre><code>{
 "input": {
 "question":"USA"
 },
 "messages": [{
 "assistant": "USA is a ..."
 }, {
 "user": "How about Mexico?"
 }]
}</code></pre>Technical<ul><li>ChatGPT</li></ul>From ChatGPT-3.5-turbo onwards they are supporting multiple messages<pre><code> {"role": "system", "content": "You are a helpful assistant."},
 {"role": "user", "content": "Who won the world series in 2020?"},
 {"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."},
 {"role": "user", "content": "Where was it played?"}</code></pre><a target="_blank" rel="noopener noreferrer nofollow" class="text-indigo-500 dark:text-indigo-300 cursor-pointer" href="https://platform.openai.com/docs/guides/gpt/chat-completions-api">https://platform.openai.com/docs/guides/gpt/chat-completions-api</a><ul><li>Llama2<img src="https://fb-usercontent.fra1.cdn.digitaloceanspaces.com/1696057312251.png"></li></ul>Consideration / Open Questions<ul><li>How to handle maximum token size?</li><li>Advance user want to customize Llama2, how can they do it?</li></ul>

Enhanced Messaging Types Support

To account for API Keys &amp; Credits consumption

[Weave] Accounting System

Name of AI model:
Qwen1.5

Huggingface / API Documentation Links: https://huggingface.co/collections/Qwen/qwen15-65c0a2f577b1ecb76d786524
https://openrouter.ai/models?q=qwen
https://www.reddit.com/r/LocalLLaMA/comments/19e9xfj/introducing_qwen/

Purpose of using this model:
Good for Mandarin and Japanese language inferencing, apparently has beaten GPT4 turbo for understanding Mandarin. Could be useful for those foreign language LLM requirements as a viable alternative to GPT. Issues involving hyperalignment as it's trained by distilling GPT lol.
Current use cases that is already using this model
Coding (surprisingly), different chatbots for different languages etc etc. Multimodal it's possible to have other input/output forms like visual and audio too.

Qwen1.5

Before user selects a model to run their prompt, the model catalog can help the users by<ol><li>reading the description of the model to help them better understand of the available model</li><li>shortlist the LLM models according to their use case and preference</li><li>searching for the model for the ease of finding their preferred model</li></ol>This model catalog is also considered as part of the expansion plan where Weave will be the aggregator of dAi models

Model Catalog

product@chasm.net

Use caseAs a user, I want to be able to search for prompts in the prompts side panel under the workflow moduleCurrent BehaviourIn workflow module, when user types in the search bar in the left side panel, it doesn’t bring up any prompts.Expected BehaviourWhen user types the name of the prompt in the search bar, it should bring up related prompts.

As a user, I want to be able to search for prompts in the prompts side panel in Workflow module

Content: A detailed explanation on how you want the feature to work and what is the use case of the feature? If possible, include any image references, videos, screenshots, or sketches to help convey your ideas more effectively. Impact of Not Having This Feature: What are the consequences of not implementing this feature. Will it hinder your or other users' ability to use the product effectively? [Example] Title: Enable weave to be used as a plugin on social media manager tools Content: As a social media manager, I'm using Later to plan and manage the social media post. Currently, I'm using Later to pre-schedule the post and use some other content generator tools to generate the contents and have to manually upload the content to Later. I find that Weave is super helpful for content generation and wish that there could be a Weave plugin on Later to create seamless content management.

[Template] A brief title that clearly describe what the feature is about

Shuwei Li

The feature is about building a LLM Listing system, whereby the listings of LLMs is managed and automated based on approval from the back office. This would allow Weave users to add any LLMs on their own. Users can add custom LLM by just inserting API url, key and documentations url. Weave BE will scrape the docs, build the manifest and store in on db.Impact of not having this featureThis feature doesn't affect it's functionality but will add value to Weave as more LLMs can be supported

Automated LLM Listing System

We should add in prompt injection prevention mechanism: • OpenAI moderation (free) undefined: <a href="https://platform.openai.com/docs/guides/moderation/overview" rel="noopener noreferrer" target="_blank">https://platform.openai.com/docs/guides/moderation/overview</a> • Prompt injection prevention from Bittensor SN14 undefined: <a href="https://docs.synapsec.ai" rel="noopener noreferrer" target="_blank">https://docs.synapsec.ai</a>Johnson got the api key from <a href="http://synapsec.ai" rel="noopener noreferrer" target="_blank">http://synapsec.ai</a> as well

Add prompt injection prevention mechanism

With weave api being open ai compatible, user can just change their API in just 2 lines, then we can onboard user that used OpenAI quickly. <pre><code>const openai = new OpenAI({ 
 baseURL: 
"https://pms.chasm.net/api/prompts/execute/44", 
 apiKey: OPENAI_API_KEY,
 });``` </code></pre>Also with this feature, we can integrate with Langchain easily as well, user can just change model through API Slack Discussion: <a target="_blank" rel="noopener noreferrer nofollow" class="text-indigo-500 dark:text-indigo-300 cursor-pointer" href="https://chasm-talk.slack.com/archives/C05PHU191GX/p1715774841176749">https://chasm-talk.slack.com/archives/C05PHU191GX/p1715774841176749</a>

Making Weave API OpenAI compatible

Currently, there is the draft feature which allows users to edit prompts while the API will always be running on the saved version. If the user decides that they want to discard their draft version and revert to the saved version, there is no button/feature for that.

Revert feature

There could be two sharing scenarios:<ol><li>Pure sharing of the workflow, where the receivers can make a copy from the workflow, and edit it as his own version</li><li>Create a collaborative team to manage the workflow together</li></ol>No 2. would be very useful, if in the future, we've more business users.

Enable users to share workflow as a template

Why do we need this?<ol><li>Perform the formatting of the output before it get stored into a variable to be used by following prompts<ul><li>String formulas - appending text? </li></ul></li><li>Extracting values of the prompt output before it get stored into a variable</li><li>Perform operations on numerical outputs</li></ol>

Method to process the output for a prompt

When a prompt produces an output, there can be a step or automated action to extract the keys into a workflow variable

[Weave] Extractable JSON from Prompt responses

Agents are super powerful to be used in LLM in order for LLM to use a certain tools.<img src="https://fb-usercontent.fra1.cdn.digitaloceanspaces.com/1696161273312.png">Some reverse engineering of Agents: <a target="_blank" rel="noopener noreferrer nofollow" class="text-indigo-500 dark:text-indigo-300 cursor-pointer" href="https://www.notion.so/hoogagaming/Agents-e26775253c5145159fe587e162432f9d?pvs=4">https://www.notion.so/hoogagaming/Agents-e26775253c5145159fe587e162432f9d?pvs=4</a>What you can achieve w Agent is that let say you ask"Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?"using LLM without Agent, it will gives u data until 2021 and maybe not be capable of doing Math, which might be wrong.With Agents (LLM Math &amp; Serp API), the prompt will be breakdown into tasks, and it will finds out through internet who's Leo's gf then get her age. After that it will use LLM Math to calculate the value.The UI can be just simple box for users to tick<img src="https://fb-usercontent.fra1.cdn.digitaloceanspaces.com/1695799969968.png">Then we can just focusing on building Agents plugin

[Weave] Agents

btw we’ve been meeting a lot of ppl in sg, I think our image prompt could be one feature we can add in, can secure a few users already hahaaaa ^ Note, lower priority compared to most PMS centric features

[Weave] Image prompt flow.

For users who sign up with Google, there is no way to update their password except for using the forgot password. It might be good to have add a UI for user to update password.

Suggest to add a Update Password UI for users who sign up with Google

Engineers

Different LLM has different set of tokenizers. To streamline the token calculation method on our platform, it's best to develop our own set of tokenisers. Reference: <a target="_blank" rel="noopener noreferrer nofollow" class="text-indigo-500 dark:text-indigo-300 cursor-pointer" href="https://github.com/huggingface/tokenizers">https://github.com/huggingface/tokenizers</a>

Chasm

Main Roadmap