Skip to main content

Error: the limit of 100,000 enabled tasks has been reached

Genevieve LaLonde
Updated

An error like the one below may be encountered when creating a data stream in RelationalAI, or executing new code.

-- Snowflake Tables Streaming Error ------------------------------------------------

Message: Error creating data stream

Error: Task DS_<HASH>_TASK cannot be resumed or created
because the limit of 100,000 enabled tasks has been reached. There are currently
100,002 enabled tasks.

------------------------------------------------------------------------------------

 

Data Stream

A data stream is how data is integrated into your RelationalAI native application from your Snowflake Account. Once the stream is created, the CDC engine is used to create an index of the data local to the RAI app. This allows quick execution of queries in RelationalAI.

Data streams are automatically created in RelationalAI when you declare your data sources in your code, and then execute that code. During execution, if there is not currently a stream created on a particular table then it will automatically be created, and that table will be automatically indexed. This is why the first run of a query on a new table is slower than subsequent runs. Anytime you query that same table in RAI after that, the query simply checks that the index is up to date (updating it if necessary), and then executes.

For more information on data streams in RAI, see our docs: https://docs.relational.ai/manage/data/manage/ 

 

Task

In Snowflake, tasks are used for automated data processing. For more information on tasks, see the Snowflake documentation: https://docs.snowflake.com/en/user-guide/tasks-intro 

For more information on troubleshooting tasks in Snowflake, see their documentation: https://docs.snowflake.cn/en/user-guide/tasks-ts 

 

Task Limit

Snowflake enforces a limit of 100,000 tasks on each Snowflake Account. Since the RelationalAI application is a part of your Snowflake account, it is also subject to that limit. This means if the total count of your tasks + RAI tasks exceeds 100,000 then it will prevent further tasks from being created or resumed.

 

RelationalAI Task Usage for Data Streams

RAI utilizes tasks for data integration, known as data streams. While the stream itself is a transaction, this transaction will have 2 batches, one for gathering metadata, and one for streaming the data. Under that, the streaming behavior uses tasks. Through these tasks, the index is built which allows RelationalAI to access your data in queries.

Altogether, this means that if there are too many tasks on your account, then RAI cannot execute any queries. This is true for any queries that would require creating a stream, or using a stream to update an index.

 

Resolution

To resolve the situation permanently, reduce the count of tasks on your Snowflake account. This requires dropping tasks or consolidating tasks.

 

 

Workarounds

Suspend Streams

In the meantime, you can prevent RelationalAI from streaming data. This will allow you to query tables that have previously streamed. It will simply use the last snapshot of the data. 

However, this will not work if you have never queried the table. That is because the stream has not yet been created. Therefore, there is no old snapshot to use in query results.

Additionally, while streaming is disabled queries will not reflect new changes in the data. This is because the stream is how those updates are brought into your RAI model. The long term solution is to reduce the task count on the account.

 

To temporarily disable streaming on a table, you can use:

CALL relationalai.api.suspend_data_stream('<db>.<schema>.<table_or_view>');

Be sure to resume streaming on the table once you have reduced the task count:

CALL relationalai.api.resume_data_stream('<db>.<schema>.<table_or_view>');

 

For more information about suspending streaming, see our doc: https://docs.relational.ai/manage/data/manage/#suspend-a-data-stream 

For more information about this command itself, see our API doc: 

https://docs.relational.ai/api/sql/api/suspend_data_stream 

 

Delete Streams

If you have a lot of data streams, another option is to delete any of them that are not in use. First check what streams there are in RelationalAI:

SELECT * FROM relationalai.api.data_streams;

In the result, note the FQ_OBJECT_NAME column. This lists the fully qualified name of the source table that the stream operates on. If you don't use that table in your code anymore, then delete that stream. Copy/paste the FQ_OBJECT_NAME into the command below and execute it to delete the stream on this table.

CALL relationalai.api.delete_data_stream('<db>.<schema>.<table_or_view>');

 

For more information about this command, see our API doc: https://docs.relational.ai/api/sql/api/delete_data_stream/ 

 

 

Related to