You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group. Recipient you are notified that disclosing, copying, distributing or taking anyĪction in reliance on the contents of this information is strictly prohibited. When I tried to start my webserver it couldnt able to start. I have configured airflow with mysql metadb with local executer. Please notify the sender immediately by e-mail if you have received this e-mailīy mistake and delete this e-mail from your system. I have installed airflow via github source. Named addressee you should not disseminate, distribute or copy this e-mail. Information and is intended only for the individual named. NOT a time in the past) and decides that it’s not ready to. When Airflow evaluates your DAG file, it interprets datetime.now() as the current timestamp (i.e. When I close the PuTTY session and reconnect to the server, I cannot find the scheduler process. For a DAG to be executed, the startdate must be a time in the past, otherwise Airflow will assume that it’s not yet ready to execute. When I run this command, the airflow scheduler starts and I can see the airflow-scheduler.pid file in my airflow home folder, but the process does not live for long.
With it are confidential and intended solely for the use of the individual orĮntity to whom they are addressed. But that’s not how Airflow reads datetime.now(). In the paragraph Task Instance Details you will see why a DAG is waiting or not running. Now select the first task and click on Task Instance. If nothing else works, you can use the web ui to click on the dag, then on Graph View. They've failed 20 times today, at last check (a couple more times since I took these screen shots). No, I trigger my DAGs manually via the REST API. If this task instance does not start soon please contact your Airflow administrator for assistance.Īlso, I am seeing the Kubernetes pods for the scheduler and worker failing constantly. The following configuration values may be limiting the number of queueable processes: parallelism, dag_concurrency, max_active_dag_runs_per_dag, non_pooled_task_slot_count To start a scheduler, simply run the command: airflow scheduler. The scheduler uses the configured Executor to run tasks that are ready. It uses the configuration specified in airflow.cfg. The scheduler is down or under heavy load To kick it off, all you need to do is execute the airflow scheduler command. In most cases this just means that the task will probably be scheduled soon unless:
AIRFLOW SCHEDULER NOT STARTING DOWNLOAD
Task instance info for each of the stuck tasks (all of the tasks) lists this as the reason:Īll dependencies are met but the task instance is not running. Download the docker-compose.yaml airflow-scheduler The scheduler monitors all tasks and DAGs, then triggers the task instances once their dependencies are. This looks like a clue to what's wrong, but I don't know where to look further on that. I tried with previous versions of Docker: 1.12.6 (14937) and 17.03.1 (16048), the problem is still the same.I've noticed that none of these tasks has a hostname listed in the table, but when I run the same DAG on my local machine, they do (it's the name of the docker image I'm running them in). It used to work well until 1-2 weeks ago, not sure what has changed (I use a modified version, but both the modified version and this docker compose have the same problem).ĭocker: Version 17.06.0-ce-mac18 (18433) Channel: stableĭocker-compose: version 1.14.0, build c7bdf9e
so this docker compose should be checked as well. It might also be a problem with Airflow itself, but if I launch it with basic configuration, it works. But with this docker compose, DAGs are not launched. The respective trademarks mentioned in the offering are owned by the respective companies, and use of them does not imply any affiliation or endorsement. basic default configuration), it works (DAGs run, and I get the log). If I use Airflow on my machine without Docker (macOS Sierra 10.12.5) by launching first the scheduler and then the webserver with SequentialExecutor (i.e. I have the same issue with 1.8.1, but in my case it seems like a consequence of #94. I left several comments in #44 about this, since both might be related.
0 kommentar(er)
