Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

[XDCR] Incomplete replication: Active key count is incorrect - build sanity error

Description

Last good build: 7.0.0-2975

Happens consistently since 2977

test:

./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False -t xdcr.biXDCR.bidirectional.load_with_async_ops,replicas=1,items=1000,ctopology=chain,rdirection=bidirection,update=C1-C2,delete=C1-C2

from console:

12:08:19 [2020-09-02 12:08:19,010] - [xdcrnewbasetests:2510] WARNING - Not Ready: vb_replica_curr_items 1050 == 1400 expected on C1, default bucket
12:08:22 [2020-09-02 12:08:22,014] - [rest_client:2230] INFO - http://172.23.105.151:8091/pools/default/buckets/default/stats?zoom=minute
12:08:22 [2020-09-02 12:08:22,113] - [xdcrnewbasetests:2510] WARNING - Not Ready: vb_replica_curr_items 1050 == 1400 expected on C1, default bucket
12:08:25 [2020-09-02 12:08:25,116] - [rest_client:2230] INFO - http://172.23.105.151:8091/pools/default/buckets/default/stats?zoom=minute
12:08:25 [2020-09-02 12:08:25,220] - [xdcrnewbasetests:2510] WARNING - Not Ready: vb_replica_curr_items 1050 == 1400 expected on C1, default bucket
12:08:28 [2020-09-02 12:08:28,223] - [xdcrnewbasetests:2514] ERROR - ERROR: Timed-out waiting for replica item count to match
12:08:28 [2020-09-02 12:08:28,226] - [remote_util:298] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5
12:08:28 [2020-09-02 12:08:28,352] - [remote_util:336] INFO - SSH Connected to 172.23.105.151 as root
12:08:28 [2020-09-02 12:08:28,734] - [remote_util:3422] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True
12:08:29 [2020-09-02 12:08:29,080] - [remote_util:3578] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
12:08:29 [2020-09-02 12:08:29,081] - [xdcrnewbasetests:2540] INFO - Executing cbvdiff for bucket default
12:08:29 [2020-09-02 12:08:29,081] - [remote_util:3269] INFO - running command.raw on 172.23.105.151: /opt/couchbase/bin/cbvdiff -b default 172.23.105.151:11210,172.23.105.153:11210
12:08:29 [2020-09-02 12:08:29,237] - [remote_util:3310] INFO - command executed successfully with root
12:08:29 [2020-09-02 12:08:29,237] - [remote_util:3104] INFO - Authentication error for default
12:08:29 [2020-09-02 12:08:29,247] - [rest_client:2230] INFO - http://172.23.107.160:8091/pools/default/buckets/default/stats?zoom=minute
12:08:29 [2020-09-02 12:08:29,346] - [xdcrnewbasetests:2484] INFO - Saw: vb_active_curr_items 1400 == 1400 expected on C2, default bucket
12:08:29 [2020-09-02 12:08:29,350] - [rest_client:2230] INFO - http://172.23.107.160:8091/pools/default/buckets/default/stats?zoom=minute
12:08:29 [2020-09-02 12:08:29,444] - [xdcrnewbasetests:2522] INFO - Saw: vb_replica_curr_items 1400 == 1400 expected on C2, default bucket
12:08:29 [2020-09-02 12:08:29,496] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default
12:08:29 [2020-09-02 12:08:29,576] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default
12:08:29 [2020-09-02 12:08:29,666] - [task:2278] INFO - 1700 items will be verified on default bucket
12:08:30 [2020-09-02 12:08:30,369] - [task:2287] INFO - 0 items were verified
12:08:30 [2020-09-02 12:08:30,483] - [task:2317] ERROR - Read failed via memcached client. Error: None
12:08:30 [<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/task.py, line 1057 in run>, <FrameSummary file lib/tasks/task.py, line 2298 in _next_>, <FrameSummary file lib/tasks/task.py, line 2320 in _check_valid_keys>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
12:08:30 Wed Sep 2 12:08:30 2020
12:08:30 [<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file testrunner.py, line 558 in run>, <FrameSummary file /usr/local/lib/python3.7/unittest/runner.py, line 176 in run>, <FrameSummary file /usr/local/lib/python3.7/unittest/case.py, line 676 in _call_>, <FrameSummary file /usr/local/lib/python3.7/unittest/case.py, line 628 in run>, <FrameSummary file pytests/xdcr/biXDCR.py, line 73 in load_with_async_ops>, <FrameSummary file pytests/xdcr/xdcrnewbasetests.py, line 3932 in verify_results>, <FrameSummary file pytests/xdcr/xdcrnewbasetests.py, line 2574 in verify_data>, <FrameSummary file lib/tasks/future.py, line 160 in result>, <FrameSummary file lib/tasks/future.py, line 111 in __get_result>]
12:08:30 [2020-09-02 12:08:30,485] - [xdcrnewbasetests:3935] ERROR - None
12:08:30 [2020-09-02 12:08:30,485] - [xdcrnewbasetests:3470] INFO - Verifying RevIds for Couchbase Cluster: C1, Master Ip: 172.23.105.151 -> Couchbase Cluster: C2, Master Ip: 172.23.107.160, bucket default
12:08:30 [2020-09-02 12:08:30,538] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default
12:08:30 [2020-09-02 12:08:30,653] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default
12:08:30 [2020-09-02 12:08:30,787] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default
12:08:30 [2020-09-02 12:08:30,888] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default
12:08:30 [2020-09-02 12:08:30,991] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default
12:08:31 [2020-09-02 12:08:31,063] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default
12:08:31 [2020-09-02 12:08:31,175] - [task:2403] INFO - RevID verification: in progress for default in scope:None in collection: None
12:08:33 [2020-09-02 12:08:33,738] - [task:2412] INFO - RevId Verification : 1400 existing items have been verified
12:08:33 [2020-09-02 12:08:33,738] - [task:2414] INFO - RevId Verification : 300 deleted items have been verified
12:08:33 [2020-09-02 12:08:33,738] - [task:2417] INFO - RevId Verification : 0 keys were apparently filtered out and not found in target bucket
12:08:33 [2020-09-02 12:08:33,745] - [rest_client:2230] INFO - http://172.23.105.151:8091/pools/default/buckets/default/stats?zoom=minute
12:08:33 [2020-09-02 12:08:33,858] - [xdcrnewbasetests:3491] ERROR - 350 keys not found on 172.23.105.151, printing first 100 keys: [('key: C2-key-649', 'vbucket: 872'), ('key: C2-key-119', 'vbucket: 848'), ('key: C2-key-482', 'vbucket: 907'), ('key: C2-key-358', 'vbucket: 959'), ('key: C2-key-263', 'vbucket: 642'), ('key: C2-key-330', 'vbucket: 830'), ('key: C2-key-260', 'vbucket: 907'), ('key: C2-key-196', 'vbucket: 822'), ('key: C2-key-302', 'vbucket: 541'), ('key: C2-key-103', 'vbucket: 670'), ('key: C2-key-470', 'vbucket: 541'), ('key: C2-key-366', 'vbucket: 810'), ('key: C2-key-352', 'vbucket: 874'), ('key: C2-key-347', 'vbucket: 539'), ('key: C2-key-668', 'vbucket: 601'), ('key: C2-key-137', 'vbucket: 734'), ('key: C2-key-23', 'vbucket: 887'), ('key: C2-key-140', 'vbucket: 1019'), ('key: C2-key-201', 'vbucket: 726'), ('key: C2-key-536', 'vbucket: 720'), ('key: C2-key-214', 'vbucket: 935'), ('key: C2-key-252', 'vbucket: 680'), ('key: C2-key-476', 'vbucket: 894'), ('key: C2-key-276', 'vbucket: 1011'), ('key: C2-key-226', 'vbucket: 644'), ('key: C2-key-325', 'vbucket: 591'), ('key: C2-key-632', 'vbucket: 1019'), ('key: C2-key-351', 'vbucket: 611'), ('key: C2-key-150', 'vbucket: 736'), ('key: C2-key-514', 'vbucket: 744'), ('key: C2-key-591', 'vbucket: 603'), ('key: C2-key-60', 'vbucket: 530'), ('key: C2-key-35', 'vbucket: 783'), ('key: C2-key-127', 'vbucket: 965'), ('key: C2-key-223', 'vbucket: 750'), ('key: C2-key-622', 'vbucket: 736'), ('key: C2-key-98', 'vbucket: 849'), ('key: C2-key-373', 'vbucket: 603'), ('key: C2-key-547', 'vbucket: 662'), ('key: C2-key-54', 'vbucket: 594'), ('key: C2-key-113', 'vbucket: 901'), ('key: C2-key-162', 'vbucket: 963'), ('key: C2-key-275', 'vbucket: 762'), ('key: C2-key-469', 'vbucket: 730'), ('key: C2-key-360', 'vbucket: 585'), ('key: C2-key-12', 'vbucket: 861'), ('key: C2-key-686', 'vbucket: 866'), ('key: C2-key-155', 'vbucket: 650'), ('key: C2-key-659', 'vbucket: 627'), ('key: C2-key-4', 'vbucket: 602'), ('key: C2-key-57', 'vbucket: 859'), ('key: C2-key-58', 'vbucket: 996'), ('key: C2-key-655', 'vbucket: 965'), ('key: C2-key-597', 'vbucket: 824'), ('key: C2-key-171', 'vbucket: 977'), ('key: C2-key-381', 'vbucket: 973'), ('key: C2-key-70', 'vbucket: 777'), ('key: C2-key-640', 'vbucket: 692'), ('key: C2-key-51', 'vbucket: 568'), ('key: C2-key-600', 'vbucket: 728'), ('key: C2-key-11', 'vbucket: 596'), ('key: C2-key-47', 'vbucket: 576'), ('key: C2-key-570', 'vbucket: 991'), ('key: C2-key-282', 'vbucket: 774'), ('key: C2-key-212', 'vbucket: 708'), ('key: C2-key-69', 'vbucket: 974'), ('key: C2-key-242', 'vbucket: 947'), ('key: C2-key-204', 'vbucket: 700'), ('key: C2-key-558', 'vbucket: 818'), ('key: C2-key-14', 'vbucket: 574'), ('key: C2-key-667', 'vbucket: 742'), ('key: C2-key-433', 'vbucket: 888'), ('key: C2-key-66', 'vbucket: 881'), ('key: C2-key-457', 'vbucket: 591'), ('key: C2-key-447', 'vbucket: 852'), ('key: C2-key-342', 'vbucket: 625'), ('key: C2-key-512', 'vbucket: 907'), ('key: C2-key-603', 'vbucket: 977'), ('key: C2-key-439', 'vbucket: 941'), ('key: C2-key-402', 'vbucket: 850'), ('key: C2-key-671', 'vbucket: 670'), ('key: C2-key-436', 'vbucket: 786'), ('key: C2-key-628', 'vbucket: 565'), ('key: C2-key-33', 'vbucket: 620'), ('key: C2-key-634', 'vbucket: 664'), ('key: C2-key-517', 'vbucket: 993'), ('key: C2-key-544', 'vbucket: 927'), ('key: C2-key-594', 'vbucket: 561'), ('key: C2-key-569', 'vbucket: 792'), ('key: C2-key-41', 'vbucket: 803'), ('key: C2-key-251', 'vbucket: 929'), ('key: C2-key-307', 'vbucket: 631'), ('key: C2-key-690', 'vbucket: 794'), ('key: C2-key-161', 'vbucket: 714'), ('key: C2-key-326', 'vbucket: 838'), ('key: C2-key-379', 'vbucket: 654'), ('key: C2-key-128', 'vbucket: 890'), ('key: C2-key-29', 'vbucket: 930'), ('key: C2-key-375', 'vbucket: 824'), ('key: C2-key-52', 'vbucket: 817')]

from xdcr log:

Change log:

- fixes related to replication to non-collection aware target
- when using DNS SRV as hostname, follow the records to determine intent
- User Intent heuristic is incorrect for full-encryption when ref does not provide port number
- convert metric type to gauge for those metrics that could drop in numbe
- bandwidth usage incorrect comment on rate unit

Components

Affects versions

Fix versions

Labels

Environment

None

Link to Log File, atop/blg, CBCollectInfo, Core dump

None

Release Notes Description

None

Attachments

11

Activity

Arunkumar Senthilnathan October 29, 2020 at 7:35 PM

Issue not seen in build sanity any more - closing this issue

CB robot September 17, 2020 at 3:29 AM

Build couchbase-server-7.0.0-3138 contains goxdcr commit 11dde25 with commit message:
- backfillMgr instantiation errors

Neil Huang September 10, 2020 at 12:36 AM

Looks like the non-active source node has some errors:

Because of this, the backfill request handler was not created... leading to inability to start the pipeline:

Pavithra Mahamani September 9, 2020 at 10:44 PM
Edited

I dont see anything wrong with the test steps. The doc count on the target does not match even after waiting for > 1 hour. Could you please take a look?

This is one set of logs from build sanity:
Logs: http://server.jenkins.couchbase.com/job/build_sanity_matrix/DISTRO=centos7,TYPE=4node/8000/
test.log:

Another set from local run:
Src cluster:


Dest cluster:

This is the test:

Test log:

Arunkumar Senthilnathan September 3, 2020 at 10:45 PM

there is one more failure in build sanity:

10:58:06 ======================================================================
10:58:06 FAIL: test_xdcr_with_filter (xdcr.advFilteringXDCR.XDCRAdvFilterTests)
10:58:06 ----------------------------------------------------------------------
10:58:06 Traceback (most recent call last):
10:58:06 File "pytests/xdcr/advFilteringXDCR.py", line 129, in test_xdcr_with_filter
10:58:06 self.verify_results()
10:58:06 File "pytests/xdcr/advFilteringXDCR.py", line 69, in verify_results
10:58:06 self.verify_filtered_items(self.src_master, self.dest_master, replications)
10:58:06 File "pytests/xdcr/xdcrnewbasetests.py", line 3821, in verify_filtered_items
10:58:06 self.filter_exp))
10:58:06 AssertionError: Doc count 1000 on 172.23.105.153 does not match doc count 499 on 172.23.107.184 after applying filter {'default': {"( REGEXP_CONTAINS(META().id,'0$') )"}}
10:58:06
10:58:06 ----------------------------------------------------------------------

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Is this a Regression?

Yes

Triage

Untriaged

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created September 3, 2020 at 12:05 AM
Updated September 16, 2024 at 7:13 AM
Resolved September 17, 2020 at 3:06 AM
Instabug