跨链交易发起：0x0bbbcf153f17ec0e1f12d698bbc64f9d242bc1cd1312dd5f34febf0e6cb6601a

1. 根据tx_hash查看表l2_to_l1_logs，得到所在的miniblock_number为15841
2. 根据上面的miniblock_number，根据number查看表miniblocks

   select * from miniblocks where number=15841;

3. 得到当前跨链交易所在l1_batch_number为7670
4. 根据l1_batch_number，以number查看l1_batches表

   select number,is_finished,eth_commit_tx_id,eth_prove_tx_id,eth_execute_tx_id from l1_batches where number=7670;

   得到

   number | is_finished | eth_commit_tx_id | eth_prove_tx_id | eth_execute_tx_id 
   --------+-------------+------------------+-----------------+-------------------
   7670 | t           |            41073 |           41148 |             41151

5. 根据eth_commit_tx_id查询eth_txs，得到CommitBlocks对应交易信息

   select nonce,contract_address,tx_type,has_failed,sent_at_block,tx_status,confirmed_eth_tx_history_id from eth_txs where id=41073 ORDER BY updated_at DESC limit 1;

   nonce |              contract_address              |   tx_type    | has_failed | sent_at_block | tx_status | confirmed_eth_tx_history_id 
    -------+--------------------------------------------+--------------+------------+---------------+-----------+-----------------------------
     41091 | 0x5e3e5f6ef0e21f0cf5b4c3acd3cf29740b1cbbd8 | CommitBlocks | f          |               | Done      |                       43194

6. 根据confirmed_eth_tx_history_id得到CommitBlocks对应交易hash

   select eth_tx_id,tx_hash,confirmed_at from eth_txs_history where id=43194  ORDER BY updated_at DESC limit 10;

   eth_tx_id |                              tx_hash                               |        confirmed_at        
    -----------+--------------------------------------------------------------------+----------------------------
         41073 | 0x0b01e199877faef52b95477119f53bf546a2915bc903132331f41542e58da53d | 2023-09-18 03:50:19.059576

7. 同理查询eth_prove_tx_id和eth_execute_tx_id得到对应的交易hash

问题背景

从一些外部消息得知

9月12日消息，据zkSync Era区块链浏览器显示，zkSync Era主网疑似出现宕机情况，zkSync Era提交给以太坊的最新批次为#208455，时间为14:14，区块高度已暂停于#13641404，已暂停出块37分钟。

跟进缘由

由于现有部分项目基于zkSync Era，所以需要确认下问题起因是什么，是否存在官方新版修复，目前现有版本会不会同样存在问题

确认问题

先从浏览器数据，确认下，是否存在消息描述问题，以及分析下问题位置（区块浏览器/链节点）

确认下Batch高度时间

208455 与 208456 相差时间符合预期

确认下Block高度时间

确认下Batch Commit时间

分析结果

从节点Batch和Block生成时间，以及Batch Commit，对比消息中的#208455前后时间差，综合来看，链方面数据无宕机，
大概率是当时区块浏览器服务方面，或者连接的某些提供数据RPC节点，出现了区块同步不及时问题。

如何避免

浏览器和提供查询的RPC节点做多个主备灾备, 实时高度状态检查和线路自动切换

所在服务器安装对应的显卡驱动（一般都有了），需要使用对应的 cuda image与其对应

对外prover docker统一使用ubuntu 20.04

• base:从 CUDA 9.0 开始，包含部署预构建 CUDA 应用程序的最低限度（libcudart）。如果您想手动选择要安装的 CUDA 软件包，请使用此映像。
• runtime:通过添加 CUDA 工具包中的所有共享库来扩展基础映像。如果您有使用多个 CUDA 库的预构建应用程序，请使用此映像。
• devel:通过添加编译器工具链、调试工具、标头和静态库来扩展运行时映像。使用此映像从源代码编译 CUDA 应用程序

数据来源：https://hub.docker.com/r/nvidia/cuda/tags

问题描述

当prover异常，或者某些原因导致各个服务不稳定时，prover_jobs 卡在in_gpu_proof 状态，跳过的证明无法自恢复进行补全。

测试数据

主节点

zksync_local=# select l1_batch_number, status, created_at from prover_jobs order by created_at desc;
 l1_batch_number |    status    |         created_at         
-----------------+--------------+----------------------------
            1147 | successful   | 2023-09-06 05:58:18.199706
            1146 | successful   | 2023-09-06 02:57:31.196466
            1145 | successful   | 2023-09-06 02:19:07.253418
            1144 | successful   | 2023-09-06 01:46:06.18188
            1143 | in_gpu_proof | 2023-09-05 11:27:41.230814
            1142 | in_gpu_proof | 2023-09-05 11:27:37.413038
            1141 | in_gpu_proof | 2023-09-05 11:27:31.75103
            1140 | in_gpu_proof | 2023-09-05 11:27:28.767585
            1139 | in_gpu_proof | 2023-09-05 11:27:23.971785

扩展节点

zksync_local=# select l1_batch_number, status, created_at from prover_jobs order by created_at desc;
 l1_batch_number |    status    |         created_at         
-----------------+--------------+----------------------------
            1147 | in_gpu_proof | 2023-09-07 10:59:47.841166
            1146 | in_gpu_proof | 2023-09-07 10:59:46.586542
            1145 | in_gpu_proof | 2023-09-07 10:59:45.218178
            1144 | in_gpu_proof | 2023-09-07 10:59:43.862459
            1143 | in_gpu_proof | 2023-09-07 10:59:42.610455
            1142 | in_gpu_proof | 2023-09-07 10:59:41.344974
            1141 | in_gpu_proof | 2023-09-07 10:59:39.956309
            1140 | in_gpu_proof | 2023-09-07 10:59:38.589833
            1139 | in_gpu_proof | 2023-09-07 10:59:37.219

跟进代码

先确认下当前zkSync是否有自恢复的逻辑

1. 查看数据操作core/lib/dal/src/prover_dal.rs，找到自恢复数据库操作requeue_stuck_jobs
2. 找到具体执行服务模块housekeeper，并以默认加载

   #[arg(
        long,
        default_value = "api,tree,eth,data_fetcher,state_keeper,witness_generator,housekeeper"
    )]
    components: ComponentsToRun,

   2023-09-08T12:43:43.561336Z  INFO zksync_core: Starting the components: [HttpApi, WsApi, ExplorerApi, Tree, EthWatcher, EthTxAggregator, EthTxManager, DataFetcher, StateKeeper, WitnessGenerator(None, BasicCircuits), WitnessGenerator(None, LeafAggregation), WitnessGenerator(None, NodeAggregation), WitnessGenerator(None, Scheduler), Housekeeper]

3. housekeeper初始化

   2023-09-08T12:43:47.244008Z  INFO zksync_core::house_keeper::periodic_job: Starting periodic job: GcsBlobCleaner with frequency: 60000 ms
   2023-09-08T12:43:47.244008Z  INFO zksync_core::house_keeper::periodic_job: Starting periodic job: WitnessGeneratorStatsReporter with frequency: 10000 ms
   2023-09-08T12:43:47.244016Z  INFO zksync_core::house_keeper::periodic_job: Starting periodic job: GpuProverQueueMonitor with frequency: 10000 ms
   2023-09-08T12:43:47.244037Z  INFO zksync_core::house_keeper::periodic_job: Starting periodic job: ProverStatsReporter with frequency: 5000 ms
   2023-09-08T12:43:47.244048Z  INFO zksync_core::house_keeper::periodic_job: Starting periodic job: WaitingToQueuedWitnessJobMover with frequency: 30000 ms
   2023-09-08T12:43:47.244050Z  INFO zksync_core::house_keeper::periodic_job: Starting periodic job: ProverJobRetryManager with frequency: 300000 ms
   2023-09-08T12:43:47.244030Z  INFO zksync_core::house_keeper::periodic_job: Starting periodic job: L1BatchMetricsReporter with frequency: 10000 ms
   2023-09-08T12:43:47.244171Z  INFO zksync_core::house_keeper::periodic_job: Starting periodic job: FriProverJobRetryManager with frequency: 30000 ms
   2023-09-08T12:43:47.244281Z  INFO zksync_core::house_keeper::periodic_job: Starting periodic job: WaitingToQueuedFriWitnessJobMover with frequency: 40000 ms
   2023-09-08T12:43:47.244293Z  INFO zksync_core::house_keeper::periodic_job: Starting periodic job: SchedulerCircuitQueuer with frequency: 40000 ms
   2023-09-08T12:43:47.244294Z  INFO zksync_core::house_keeper::periodic_job: Starting periodic job: WitnessGeneratorStatsReporter with frequency: 10000 ms
   2023-09-08T12:43:47.244300Z  INFO zksync_core::house_keeper::periodic_job: Starting periodic job: FriWitnessGeneratorJobRetryManager with frequency: 30000 ms
   2023-09-08T12:43:47.244296Z  INFO zksync_core::house_keeper::periodic_job: Starting periodic job: FriProverStatsReporter with frequency: 30000 ms

   此时zksync-server日志循环等待新证明
   2023-09-08T12:43:48.092403Z  INFO zksync_core::eth_sender::opside_send_plug::manage_proof_plug: submit_pending_proofs finish
   2023-09-08T12:43:48.092417Z  INFO zksync_core::eth_sender::opside_send_plug::manage_proof_plug: manage proof start try_fetch_proof_to_send
   2023-09-08T12:43:48.092433Z  INFO zksync_core::eth_sender::opside_send_plug::manage_proof_plug: manage proof: process_resend receiver for start signal
   2023-09-08T12:43:48.445424Z  INFO zksync_core::eth_sender::opside_send_plug::manage_proof_plug: try_fetch_proof_to_send: check dest block:681 commit
   2023-09-08T12:44:08.927334Z  INFO zksync_core::eth_sender::opside_send_plug::manage_proof_plug: sequence_batch_info.block_number == 0 at batch:681

4. 判断housekeeper检查服务是否正常被执行

   async fn run_routine_task(&mut self) {
    let stuck_jobs = self
        .prover_connection_pool
        .access_storage()
        .await
        .prover_dal()
        .requeue_stuck_jobs(self.processing_timeout, self.max_attempts)
        .await;
    let job_len = stuck_jobs.len();
    for stuck_job in stuck_jobs {
        vlog::info!("re-queuing prover job {:?}", stuck_job);
    }
    // vlog::info!("server.prover.requeued_jobs{}", job_len as u64);
    // vlog::info!("server.prover.self.processing_timeout{}", self.processing_timeout.as_secs() as u64);
    // vlog::info!("server.prover.self.max_attempts{}", self.max_attempts as u64);
    metrics::counter!("server.prover.requeued_jobs", job_len as u64);
   }

   self.processing_timeout ： 2700s （45分钟）
   self.max_attempts： 1

   经过测试，run_routine_task被正常定期执行，但是没有执行任务

   2023-09-08T13:13:17.988849Z  INFO zksync_core::house_keeper::prover_job_retry_manager: server.prover.requeued_jobs0
   2023-09-08T13:13:17.988864Z  INFO zksync_core::house_keeper::prover_job_retry_manager: server.prover.self.processing_timeout2700
   2023-09-08T13:13:17.988873Z  INFO zksync_core::house_keeper::prover_job_retry_manager: server.prover.self.max_attempts1

5. 查看run_routine_task具体数据库操作

   UPDATE prover_jobs
   SET status = 'queued', attempts = attempts + 1, updated_at = now(), processing_started_at = now()
   WHERE (status = 'in_progress' AND  processing_started_at <= now() - $1::interval AND attempts < $2)
   OR (status = 'in_gpu_proof' AND  processing_started_at <= now() - $1::interval AND attempts < $2)
   OR (status = 'failed' AND attempts < $2)
   RETURNING id, status, attempts

   对照681数据
   

总结分析

对于zkSync本身存在自恢复逻辑housekeeper->run->run_routine_task->requeue_stuck_jobs
然后根据self.processing_timeout, self.max_attempts两个配置参数，进行数据库查询

self.processing_timeout ： 2700s （45分钟）
self.max_attempts： 1

UPDATE prover_jobs
SET status = 'queued', attempts = attempts + 1, updated_at = now(), processing_started_at = now()
WHERE (status = 'in_progress' AND  processing_started_at <= now() - $1::interval AND attempts < $2)
OR (status = 'in_gpu_proof' AND  processing_started_at <= now() - $1::interval AND attempts < $2)
OR (status = 'failed' AND attempts < $2)
RETURNING id, status, attempts

初始任务状态为 queued,当任务成功下发到prover时，状态变更为in_gpu_proof，如果当前任务没有在设置时间内processing_timeout完成，则将当前状态改回 queued，重新下发prover(可能是其他prover)
注意：当状态变更为in_gpu_proof时，attempts 会加1
问题：由于默认max_attempts等于1，所以无法进行run_routine_task
解决：增大max_attempts，5
同时需要注意processing_timeout，应大于支持最低显卡类型，单显卡，运行的时间，否则可能导致算力浪费。

PROVER_NON_GPU_MAX_ATTEMPTS=5
PROVER_NON_GPU_GENERATION_TIMEOUT_IN_SECS=3600

方案验证

修改环境变量后

2023-09-08T14:00:44.990770Z  INFO zksync_core::house_keeper::prover_job_retry_manager: re-queuing prover job StuckProverJobs { id: 1149, status: "queued", attempts: 2 }

1. 相应in_gpu_proof任务状态更新为queued
2. 链接的prover开始proof生成

异常场景

注意：无法回滚已执行的高度，否则报Attempt to revert already executed blocks
此时，强制停止 prover，并发送交易， 制造落后场景

查询最新数据

zk f cargo run --bin block_reverter --release -- print-suggested-values --json

返回数据中包含几个信息

• committed 最新高度
• verified 已验证高度
• executed 本地已执行高度
• last_executed_l1_batch_number： 推荐回滚高度
• nonce: 执行交易nonce
• priority_fee: 推荐fee

  2023-09-13T10:25:08.901399Z  INFO zksync_core::block_reverter: Last L1 batch numbers on contract: committed 1171, verified 1170, executed 1170
  {"last_executed_l1_batch_number":1170,"nonce":3867,"priority_fee":1000000000}

  此时链上最新高度为committed：1171

测试回滚

先停止下zksync_server/zksync_external_node

合约回滚

1171回滚到1170

cd $ZKSYNC_HOME && zk f cargo run --bin block_reverter --release -- send-eth-transaction --l1-batch-number 1170 --nonce 3871  --priority-fee-per-gas 1000000000

此时会向合约发送revertBlocks交易，无需手动执行合约
此时再次查询

zk f cargo run --bin block_reverter --release -- print-suggested-values --json

此时链上最新高度为committed：1170

本地数据库回滚

注意：需要先停止下zksync_server/zksync_external_node, 否则报 ./db/main/tree/LOCK

cd $ZKSYNC_HOME && zk f cargo run --bin block_reverter --release -- rollback-db --l1-batch-number 1170 --rollback-postgres --rollback-tree --rollback-sk-cache

附加

core/bin/zksync_core/src/bin/block_reverter.rs
Command

• print-suggested-values
  显示要使用的建议值。
• send-eth-transaction
  将恢复事务发送到 L1。
• rollback-db
  将内部数据库状态恢复到前一个块。
• clear-failed-transactions
  清除失败的 L1 事务。

测试用例

core/tests/revert-test/tests/revert-and-restart.test.ts
const executedProcess = await utils.exec(
    'cd $ZKSYNC_HOME && ' +
        'RUST_LOG=off cargo run --bin block_reverter --release -- print-suggested-values --json'
    // ^ Switch off logs to not pollute the output JSON
);
const suggestedValuesOutput = executedProcess.stdout;
const { lastL1BatchNumber, nonce, priorityFee } = parseSuggestedValues(suggestedValuesOutput);
expect(lastL1BatchNumber < blocksCommittedBeforeRevert, 'There should be at least one block for revert').to.be
    .true;

console.log(
    Reverting with parameters: last unreverted L1 batch number: ${lastL1BatchNumber}, nonce: ${nonce}, priorityFee: ${priorityFee}
);

console.log('Sending ETH transaction..');
await utils.spawn(
    cd $ZKSYNC_HOME && cargo run --bin block_reverter --release -- send-eth-transaction --l1-batch-number ${lastL1BatchNumber} --nonce ${nonce} --priority-fee-per-gas ${priorityFee}
);

console.log('Rolling back DB..');
await utils.spawn(
    cd $ZKSYNC_HOME && cargo run --bin block_reverter --release -- rollback-db --l1-batch-number ${lastL1BatchNumber} --rollback-postgres --rollback-tree --rollback-sk-cache
);

let blocksCommitted = await mainContract.getTotalBlocksCommitted();
expect(blocksCommitted.eq(lastL1BatchNumber), 'Revert on contract was unsuccessful').to.be.true;