id stringlengths 15 25 | source stringclasses 1
value | chunk_index int64 0 393k | token_count int32 449 512 | text stringlengths 203 7.95k |
|---|---|---|---|---|
combine_chunk_0 | combine.txt | 0 | 512 |
<|file|>airflow/providers/opsgenie/README.rst
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache ... |
combine_chunk_1 | combine.txt | 1 | 512 | 5``
========================================== ==================
The changelog for the provider package can be found in the
`changelog <https://airflow.apache.org/docs/apache-airflow-providers-opsgenie/5.11.1/changelog.html>`_.
<|file|>airflow/providers/opsgenie/docs/index.rst
.. Licensed to the Apache Software F... |
combine_chunk_2 | combine.txt | 2 | 512 | WILL BE OVERWRITTEN AT RELEASE TIME!
.. toctree::
:hidden:
:maxdepth: 1
:caption: Commits
Detailed list of commits <commits>
apache-airflow-providers-opsgenie package
------------------------------------------------------
`Opsgenie <https://www.atlassian.com/software/opsgenie>`__
Release: 5.11.... |
combine_chunk_3 | combine.txt | 3 | 512 | _airflow_providers_opsgenie-5.11.1-py3-none-any.whl.sha512>`__)
<|file|>airflow/providers/opsgenie/docs/operators/index.rst
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regar... |
combine_chunk_4 | combine.txt | 4 | 512 | include:: /../../opsgenie/tests/system/opsgenie/example_opsgenie_alert.py
:language: python
:start-after: [START howto_opsgenie_create_alert_operator]
:end-before: [END howto_opsgenie_create_alert_operator]
.. _howto/operator:OpsgenieCloseAlertOperator:
OpsgenieCloseAlertOperator
=========================... |
combine_chunk_5 | combine.txt | 5 | 512 | under the License.
.. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
.. IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
`PROVIDER_COMMITS_TEMPLATE.rst.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
.. THE REMAINDER OF THE FILE IS AUTOMATICALLY GENER... |
combine_chunk_6 | combine.txt | 6 | 512 | required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.... |
combine_chunk_7 | combine.txt | 7 | 512 | may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the ... |
combine_chunk_8 | combine.txt | 8 | 512 | . You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the... |
combine_chunk_9 | combine.txt | 9 | 512 | NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache... |
combine_chunk_10 | combine.txt | 10 | 512 | ctree::
:hidden:
:maxdepth: 1
:caption: Resources
Example Dags <https://github.com/apache/airflow/tree/providers-neo4j/|version|/providers/neo4j/tests/system/neo4j>
PyPI Repository <https://pypi.org/project/apache-airflow-providers-neo4j/>
Installing from sources <installing-providers-from-sour... |
combine_chunk_11 | combine.txt | 11 | 512 | .tar.gz.sha512>`__)
* `The apache-airflow-providers-neo4j 3.13.1 wheel package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_neo4j-3.13.1-py3-none-any.whl>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_neo4j-3.13.1-py3-none-any.whl.asc>`__, `sha512 <https://... |
combine_chunk_12 | combine.txt | 12 | 512 | : string
- Neo4j user password
* - Port: int
- Neo4j port
.. exampleinclude:: /../../neo4j/tests/system/neo4j/example_neo4j.py
:language: python
:dedent: 4
:start-after: [START run_query_neo4j_operator]
:end-before: [END run_query_neo4j_operator]
Passing parameters into the query
^^^^^^^^... |
combine_chunk_13 | combine.txt | 13 | 512 | under the License.
.. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
.. IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
`PROVIDER_COMMITS_TEMPLATE.rst.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
.. THE REMAINDER OF THE FILE IS AUTOMATICALLY GENER... |
combine_chunk_14 | combine.txt | 14 | 512 | required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.... |
combine_chunk_15 | combine.txt | 15 | 512 |
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.... |
combine_chunk_16 | combine.txt | 16 | 512 | ="example_neo4j_sensor_basic",
start_date=days_ago(1),
schedule=None,
catchup=False,
):
wait_person_exists = Neo4jSensor(
task_id="wait_person_exists",
neo4j_conn_id="neo4j_default",
cypher="MATCH (p:Person) RETURN count(p) > 0",
)
In this example,... |
combine_chunk_17 | combine.txt | 17 | 512 | applies ``operator.itemgetter(0)`` to the first row of the
result set, effectively selecting the first value of the first row. You can override
this with a custom ``selector`` callable.
The ``selector`` receives the first row as a tuple and must return a single value
to be used by ``success`` / ``failure`` or for tru... |
combine_chunk_18 | combine.txt | 18 | 512 | be evaluated. Defaults to selecting the first
element of the row with ``operator.itemgetter(0)``.
``fail_on_empty``
When set to ``True``, the sensor raises an exception if the query returns
no rows. When ``False`` (default), the sensor simply returns ``False`` and
will poke again later.
``**kwargs``
... |
combine_chunk_19 | combine.txt | 19 | 512 | THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
.. IF YOU WANT TO MODIFY TEMPLATE FOR THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
``PROVIDER_README_TEMPLATE.rst.jinja2`` IN the ``dev/breeze/src/airflow_breeze/templates`` DIRECTORY
Package ``apache-airflow-providers-microsoft-winrm``
Release: ``3.15.1... |
combine_chunk_20 | combine.txt | 20 | 512 | .org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions a... |
combine_chunk_21 | combine.txt | 21 | 512 | below.
Requirements
------------
The minimum Apache Airflow version supported by this provider distribution is ``2.11.0``.
========================================== ==================
PIP package Version required
========================================== ==================
``apac... |
combine_chunk_22 | combine.txt | 22 | 512 | .apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permis... |
combine_chunk_23 | combine.txt | 23 | 512 | regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by app... |
combine_chunk_24 | combine.txt | 24 | 512 | to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use th... |
combine_chunk_25 | combine.txt | 25 | 512 | ``>=2.3.5; python_version < "3.14"``
``pymssql`` ``>=2.3.13; python_version >= "3.14"``
``methodtools`` ``>=0.4.7``
========================================== ======================================
Optional cross provider package dependencies
--------------... |
combine_chunk_26 | combine.txt | 26 | 512 | Changelog <changelog>
Security <security>
.. toctree::
:hidden:
:maxdepth: 1
:caption: Guides
Connection types <connections/mssql>
Operators <operators>
.. toctree::
:hidden:
:maxdepth: 1
:caption: References
Python API <_api/airflow/providers/microsoft/mssql/index>
Dial... |
combine_chunk_27 | combine.txt | 27 | 512 | 2.3.5; python_version < "3.14"``
``pymssql`` ``>=2.3.13; python_version >= "3.14"``
``methodtools`` ``>=0.4.7``
========================================== ======================================
Optional cross provider package dependencies
-------------------... |
combine_chunk_28 | combine.txt | 28 | 512 | _mssql-4.7.3-py3-none-any.whl.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_microsoft_mssql-4.7.3-py3-none-any.whl.sha512>`__)
<|file|>airflow/providers/microsoft/mssql/docs/commits.rst
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor l... |
combine_chunk_29 | combine.txt | 29 | 512 | with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either exp... |
combine_chunk_30 | combine.txt | 30 | 512 | -connection-properties?view=sql-server-ver15>`_.
When specifying the connection as URI (in :envvar:`AIRFLOW_CONN_{CONN_ID}` variable) you should specify it
following the standard syntax of DB connections - where extras are passed as parameters
of the URI. Note that all components of the URI should be URL-encoded.
For... |
combine_chunk_31 | combine.txt | 31 | 512 | connect and that's why dialects where introduced.
The default :class:`~airflow.providers.common.sql.dialects.dialect.Dialect` class has the following operations
available which underneath use SQLAlchemy to execute, but can be overloaded with specialized implementations
per database:
- ``placeholder`` specifies the d... |
combine_chunk_32 | combine.txt | 32 | 512 | or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License... |
combine_chunk_33 | combine.txt | 33 | 512 | to_guide_create_table_mssql_from_external_file]
Your ``dags/create_table.sql`` should look like this:
.. code-block::sql
-- create Users table
CREATE TABLE Users (
user_id INT NOT NULL IDENTITY(1,1) PRIMARY KEY,
username TEXT,
description TEXT
);
Inserting data into a MSSQL... |
combine_chunk_34 | combine.txt | 34 | 512 | "License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, ... |
combine_chunk_35 | combine.txt | 35 | 512 | file|>airflow/providers/microsoft/azure/README.rst
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apa... |
combine_chunk_36 | combine.txt | 36 | 512 | http`` ``>=3.14.0``
``azure-batch`` ``>=15.1.0``
``azure-ai-projects`` ``>=2.2.0``
``azure-cosmos`` ``>=4.15.0``
``azure-mgmt-cosmosdb`` ``>=9.9.0``
``azure-datalake-store`` ... |
combine_chunk_37 | combine.txt | 37 | 512 | msal-extensions`` ``>=1.3.0``
========================================== =====================
Optional cross provider package dependencies
--------------------------------------------
Those are dependencies that might be needed in order to use all the features of the package.
You need to ins... |
combine_chunk_38 | combine.txt | 38 | 512 | file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http:/... |
combine_chunk_39 | combine.txt | 39 | 512 | message in Azure Service Bus.
.. exampleinclude:: /../tests/system/microsoft/azure/example_event_schedule_asb.py
:language: python
:start-after: [START howto_trigger_asb_message_queue]
:end-before: [END howto_trigger_asb_message_queue]
How it works
------------
1. **Azure Service Bus Message Queue Trigg... |
combine_chunk_40 | combine.txt | 40 | 512 | this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software di... |
combine_chunk_41 | combine.txt | 41 | 512 | :caption: Commits
Detailed list of commits <commits>
apache-airflow-providers-microsoft-azure package
------------------------------------------------------
`Microsoft Azure <https://azure.microsoft.com/>`__
Release: 15.3.1
Provider package
----------------
This package is for the ``microsoft.azure`` provi... |
combine_chunk_42 | combine.txt | 42 | 512 | usto-data`` ``>=4.1.0,!=5.0.0``
``azure-mgmt-datafactory`` ``>=10.0.0``
``azure-mgmt-containerregistry`` ``>=15.0.0``
``azure-mgmt-compute`` ``>=38.0.0``
``azure-mgmt-containerinstance`` ``>=10.1.0``
``msgraph-core`` ... |
combine_chunk_43 | combine.txt | 43 | 512 | -sftp>`_ ``sftp``
======================================================================================================================== ====================
Optional dependencies
---------------------
These extras install optional third-party libraries that enable additional features of t... |
combine_chunk_44 | combine.txt | 44 | 512 | See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License a... |
combine_chunk_45 | combine.txt | 45 | 512 | ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
... |
combine_chunk_46 | combine.txt | 46 | 512 | : [START howto_operator_powerbi_workspaces_info]
:end-before: [END howto_operator_powerbi_workspaces_info]
Below is an example of using this operator to refresh PowerBI dataset.
.. exampleinclude:: /../tests/system/microsoft/azure/example_powerbi.py
:language: python
:dedent: 0
:start-after: [START ho... |
combine_chunk_47 | combine.txt | 47 | 512 | abric/articles/using-fabric-apis/>`__
<|file|>airflow/providers/microsoft/azure/docs/operators/ai_agents.rst
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ... |
combine_chunk_48 | combine.txt | 48 | 512 | .py
:language: python
:dedent: 4
:start-after: [START howto_operator_azure_ai_agent_create]
:end-before: [END howto_operator_azure_ai_agent_create]
.. _howto/operator:UpdateAzureAIAgentOperator:
UpdateAzureAIAgentOperator
--------------------------
Azure AI Hosted agent updates are published as new i... |
combine_chunk_49 | combine.txt | 49 | 512 | to_operator_azure_ai_agent_run]
.. _howto/operator:DeleteAzureAIAgentOperator:
DeleteAzureAIAgentOperator
--------------------------
To delete an Azure AI Hosted agent and all of its versions, use the
:class:`~airflow.providers.microsoft.azure.operators.ai_agents.DeleteAzureAIAgentOperator`.
Pass ``agent_version`` t... |
combine_chunk_50 | combine.txt | 50 | 512 | AI Search <https://learn.microsoft.com/en-us/azure/search/>`__ is the full-text and vector
store behind retrieval-augmented generation on Azure. The
:class:`~airflow.providers.microsoft.azure.hooks.ai_search.AzureAISearchHook` reads and writes the
documents of one index, from an async task: every method is a coroutine... |
combine_chunk_51 | combine.txt | 51 | 512 |
sortable and filterable key (``order_by=["id"]``) and filter on ranges of it
(``filter="id gt '<last key read>'"``), one range per call, as described in
`Paging through a large number of results <https://learn.microsoft.com/en-us/azure/search/search-pagination-page-layout#paging-through-a-large-number-of-results>`__.
... |
combine_chunk_52 | combine.txt | 52 | 512 | .. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); y... |
combine_chunk_53 | combine.txt | 53 | 512 | to ``api.powerbi.com`` too.
The triggerer still polls the refresh.
* The refresh is not cancelled when polling it fails, so the retry can reconnect to it.
Set ``durable=False`` to keep the earlier behaviour: the trigger starts the refresh, so only the triggerer
talks to Power BI, every try starts a new refresh, and... |
combine_chunk_54 | combine.txt | 54 | 512 | "); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT W... |
combine_chunk_55 | combine.txt | 55 | 512 | .
.. exampleinclude:: /../tests/system/microsoft/azure/example_adf_run_pipeline.py
:language: python
:dedent: 0
:start-after: [START howto_operator_adf_run_pipeline_async]
:end-before: [END howto_operator_adf_run_pipeline_async]
Also you can use deferrable mode in :class:`~airflow.... |
combine_chunk_56 | combine.txt | 56 | 512 | software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
Azure Synapse Operators
=======================
Azu... |
combine_chunk_57 | combine.txt | 57 | 512 | errable_flag]
:end-before: [END howto_operator_azure_synapse_run_pipeline_with_deferrable_flag]
Reference
---------
For further information, please refer to the Microsoft documentation:
* `Azure Synapse Analytics Documentation <https://docs.microsoft.com/en-us/azure/synapse-analytics//>`__
<|file|>airflow/p... |
combine_chunk_58 | combine.txt | 58 | 512 | :start-after: [START howto_operator_adls_delete]
:end-before: [END howto_operator_adls_delete]
.. _howto/operator:ADLSListOperator:
ADLSListOperator
----------------------------------
Use the
:class:`~airflow.providers.microsoft.azure.operators.adls.ADLSListOperator` to list all
file(s) from Azure DataLake Stora... |
combine_chunk_59 | combine.txt | 59 | 512 | :`~airflow.providers.microsoft.azure.operators.compute.AzureVirtualMachineStartOperator`
to start an Azure Virtual Machine.
Below is an example of using this operator to start a VM:
.. exampleinclude:: /../tests/system/microsoft/azure/example_azure_compute.py
:language: python
:dedent: 4
:start-after: [ST... |
combine_chunk_60 | combine.txt | 60 | 512 | azure.microsoft.com/en-us/products/virtual-machines/>`__
<|file|>airflow/providers/microsoft/azure/docs/operators/asb.rst
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regardi... |
combine_chunk_61 | combine.txt | 61 | 512 | operators.asb.AzureServiceBusSendMessageOperator`.
Below is an example of using this operator to execute an Azure Service Bus Send Message to Queue.
.. exampleinclude:: /../tests/system/microsoft/azure/example_azure_service_bus.py
:language: python
:dedent: 4
:start-after: [START howto_operator_send_messa... |
combine_chunk_62 | combine.txt | 62 | 512 | to_operator_create_service_bus_topic]
:end-before: [END howto_operator_create_service_bus_topic]
.. _howto/operator:AzureServiceBusTopicDeleteOperator:
Delete Azure Service Bus Topic
======================================
To Delete the Azure service bus topic you can use
:class:`~airflow.providers.microsoft.azur... |
combine_chunk_63 | combine.txt | 63 | 512 | .operators.asb.ASBReceiveSubscriptionMessageOperator`.
Below is an example of using this operator to execute an Azure Service Bus Receive Subscription Message.
.. exampleinclude:: /../tests/system/microsoft/azure/example_azure_service_bus.py
:language: python
:dedent: 4
:start-after: [START howto_operator... |
combine_chunk_64 | combine.txt | 64 | 512 | /../tests/system/microsoft/azure/example_azure_batch_operator.py
:language: python
:dedent: 0
:start-after: [START howto_azure_batch_operator]
:end-before: [END howto_azure_batch_operator]
Below is an example of using this operator to trigger a task on Azure Batch with the deferrable flag,
so that pol... |
combine_chunk_65 | combine.txt | 65 | 512 | .. exampleinclude:: /../tests/system/microsoft/azure/example_azure_analysis_services.py
:language: python
:dedent: 4
:start-after: [START howto_operator_azure_analysis_services_refresh]
:end-before: [END howto_operator_azure_analysis_services_refresh]
Wait for the refresh to finish
--------------------... |
combine_chunk_66 | combine.txt | 66 | 512 | copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law o... |
combine_chunk_67 | combine.txt | 67 | 512 | *
<|file|>airflow/providers/microsoft/azure/docs/filesystems/msgraph.rst
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this f... |
combine_chunk_68 | combine.txt | 68 | 512 | grant flow
* **password**: Password for password grant flow
* **certificate_path**: File path to a PEM certificate
* **certificate_data**: PEM certificate data
* **authority**: Microsoft Entra authority host
* **disable_instance_discovery**: Disable Microsoft Entra instance discovery
Connection extra field configurat... |
combine_chunk_69 | combine.txt | 69 | 512 | Generated report data\n")
f.write(f"Created at: {datetime.now()}\n")
Drive Discovery
^^^^^^^^^^^^^^^
When you need to find the correct drive ID for your URLs, you can use the Microsoft Graph API operators:
.. code-block:: python
from airflow.providers.microsoft.azure.operators.msgraph import MSGraphAsyn... |
combine_chunk_70 | combine.txt | 70 | 512 |
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either ex... |
combine_chunk_71 | combine.txt | 71 | 512 |
dag_failure_notification = MSGraphNotifier(
from_email="airflow@example.com",
to="team@example.com",
subject="Dag {{ dag.dag_id }} failed",
html_content="Task {{ ti.task_id }} failed on {{ ds }}",
)
with DAG(
dag_id="mydag",
schedule="@daily",
on_fa... |
combine_chunk_72 | combine.txt | 72 | 512 | at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific lang... |
combine_chunk_73 | combine.txt | 73 | 512 |
:class:`~airflow.providers.amazon.aws.transfers.gcs_to_s3.GCSToS3Operator` (``flatten_structure`` wins when both apply).
Object keys ending with ``/`` (GCS console folder markers) are not copied.
Example:
.. code-block:: python
copy_gcs_to_azure = GCSToAzureBlobStorageOperator(
task_id="gcs_to_azure_blo... |
combine_chunk_74 | combine.txt | 74 | 512 | Login and Password.
Please follow Azure
`instructions <https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal>`_
to do it.
See following example.
Set values for these fields:
.. code-block::
SFTP Conn Id: sftp_default
WASB Conn Id: wasb_default
.. contents::
:dept... |
combine_chunk_75 | combine.txt | 75 | 512 | ^^^^^^^^^^
.. include:: /operators/_partials/prerequisite_tasks.rst
.. _howto/operator:LocalFilesystemToADLSOperator:
LocalFilesystemToADLSOperator
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
:class:`~airflow.providers.microsoft.azure.transfers.local_to_adls.LocalFilesystemToADLSOperator` allows you to
upload data from loc... |
combine_chunk_76 | combine.txt | 76 | 512 | Storage Service (S3) to Azure Blob Storage.
Prerequisite Tasks
------------------
.. include:: ../operators/_partials/prerequisite_tasks.rst
Operators
---------
.. _howto/operator:S3ToWasbOperator:
Transfer Data from Amazon S3 to Blob Storage
============================================
To copy data from an Ama... |
combine_chunk_77 | combine.txt | 77 | 512 | Azure Blob Storage API documentation <https://docs.microsoft.com/en-us/rest/api/storageservices/blob-service-rest-api>`_.
This page shows how to upload data from local filesystem to Azure Blob Storage.
.. _howto/operator:LocalFilesystemToWasbOperator:
LocalFilesystemToWasbOperator
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~... |
combine_chunk_78 | combine.txt | 78 | 512 | Airflow connection.
4. Use managed identity by setting ``managed_identity_client_id``, ``workload_identity_tenant_id`` (under the hook, it uses DefaultAzureCredential_ with these arguments)
5. Fallback on `DefaultAzureCredential`_
This includes a mechanism to try different options to authenticate: Managed System Id... |
End of preview. Expand in Data Studio
Python and Docs Training Dataset
A massive, multi-format code and documentation corpus built from 70+ major open-source Python repositories (including the Pip, Keras, Gradio, Transformers, Datasets, and more).
Dataset Structure
The dataset is processed using exact BPE token chunking (512-token chunks with zero overlap via OpenAI's cl100k_base tokenizer) while preserving original Python indentation, newlines, and <|file|>path document boundaries—eliminating boundary duplication and allowing direct fixed-window pre-training without truncation or padding.
Each record in the JSONL and Parquet shards contains:
id: Unique sequential identifier for the 512-token chunk (e.g.,combine_chunk_0).source: Originating combined corpus file (combine.txt).chunk_index: Zero-based sequential index preserving repository-grouped ordering (Docs -> Code).token_count: Exact BPE token length (512tokens per row;< 512only on the final EOF tail row if retained).content: Raw, indentation-preserved block of Python code, type stubs, or documentation separated by<|file|>pathheaders.
(An accompanying flat binary shard Python_pretrain.bin stores the same continuous token stream as headerless uint32 token IDs for zero-copy np.memmap training.)
Corpus Breakdown
- Combined Repository Corpus (
combine.txt): 22,551,967 lines (~850.3 MB / ~220M+cl100k_basetokens) curated across 74 permissive-license (MIT, Apache-2.0, BSD-3) repositories, with line lengths capped at999characters and test/vendored directories excluded. - Documentation (
.md,.rst): Repository guides, API references, and technical manuals grouped ahead of each library's source tree. - Python Source & Type Stubs (
.py,.pyi): Production library implementations, core modules, and type annotations across foundational Python, data science, web, and ML frameworks (including CPython, PyTorch, TensorFlow, NumPy, SciPy, Pandas, Django, FastAPI/Starlette, and Transformers).
License
Distributed under the Apache-2.0 open-source license.
- Downloads last month
- 19