Bug #57206
closedceph_test_libcephfs_reclaim crashes during test
0%
Description
/a/vshankar-2022-08-18_04:30:42-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6978421
Core is at: ./remote/smithi061/coredump/1660821251.63191.core
file ./remote/smithi061/coredump/1660821251.63191.core ./remote/smithi061/coredump/1660821251.63191.core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_libcephfs_reclaim', real uid: 1000, effective uid: 1000, real gid: 1267, effective gid: 1267, execfn: '/usr/bin/ceph_test_libcephfs_reclaim', platform: 'x86_64'
(Didn't go to fetch the backtrace from the core)
Updated by Venky Shankar over 1 year ago
- Status changed from New to Triaged
- Assignee set to Tamar Shacked
- Labels (FS) task(easy) added
Updated by Tamar Shacked over 1 year ago
I"ve used https://github.com/ceph/ceph/blob/main/src/script/ceph-debug-docker.sh for deploying the build on container.
This is the bt of the crash, it happen in the beginning and seems to be related to parameters to 'rbd_features_from_string(const std::string& orig_value,std::ostream *err)'
I need to figure how to get symbols of /usr/lib/ceph/libceph-common.so.2 for watching rbd_features_from_string parameters
(gdb) bt full #0 0x00007fc83f6053ee in std::locale::operator==(std::locale const&) const () from /usr/lib/ceph/libceph-common.so.2 No symbol table info available. #1 0x00007fc83f4c6798 in boost::detail::lcast_ret_unsigned<std::char_traits<char>, unsigned long, char>::convert() () from /usr/lib/ceph/libceph-common.so.2 No symbol table info available. #2 0x00007fc83f4c59e4 in librbd::rbd_features_from_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::ostream*) () from /usr/lib/ceph/libceph-common.so.2 No symbol table info available. #3 0x00007fc83f150e0f in ?? () from /usr/lib/ceph/libceph-common.so.2 No symbol table info available. #4 0x00007fc83f0acfb6 in Option::pre_validate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const () from /usr/lib/ceph/libceph-common.so.2 No symbol table info available. #5 0x00007fc83f0af5cb in Option::parse_value(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::variant<std::monostate, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long, long, double, bool, entity_addr_t, entity_addrvec_t, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000l> >, Option::size_t, uuid_d>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const () from /usr/lib/ceph/libceph-common.so.2 No symbol table info available. #6 0x00007fc83f07efa2 in md_config_t::_set_val(ConfigValues&, ConfigTracker const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Option const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) () from /usr/lib/ceph/libceph-common.so.2 No symbol table info available. #7 0x00007fc83f07f417 in md_config_t::set_val_default(ConfigValues&, ConfigTracker const&, std::basic_string_view<char, std::char_traits<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /usr/lib/ceph/libceph-common.so.2 No symbol table info available. #8 0x00007fc83f08ef0a in md_config_t::md_config_t(ConfigValues&, ConfigTracker const&, bool) () from /usr/lib/ceph/libceph-common.so.2 No symbol table info available. #9 0x00007fc83f035bdc in ceph::common::CephContext::CephContext(unsigned int, ceph::common::CephContext::create_options const&) () from /usr/lib/ceph/libceph-common.so.2 No symbol table info available. #10 0x00007fc83f036c61 in ceph::common::CephContext::CephContext(unsigned int, code_environment_t, int) () from /usr/lib/ceph/libceph-common.so.2 No symbol table info available. #11 0x00007fc83f075675 in common_preinit(CephInitParameters const&, code_environment_t, int) () from /usr/lib/ceph/libceph-common.so.2 No symbol table info available. #12 0x00007fc83fe4a7f9 in ceph_create () from /lib/libcephfs.so.2 No symbol table info available. #13 0x0000555a90b7882a in update_root_mode () at ./src/test/libcephfs/reclaim.cc:149 admin = 0xe90000007f r = <optimized out> admin = <optimized out> r = <optimized out> #14 main (argc=<optimized out>, argv=0x7ffc0ba75948) at ./src/test/libcephfs/reclaim.cc:149 r = <optimized out>
Updated by Tamar Shacked over 1 year ago
The same crash reported for rgw - https://tracker.ceph.com/issues/57050
I"ll go over it to get the details.
Updated by Venky Shankar over 1 year ago
Tamar,
Were you able to go through the changes for the rgw fix here: https://github.com/ceph/ceph/pull/47504 to see if we'd need to do something similar?
Cheers,
Venky
Updated by Venky Shankar over 1 year ago
- Assignee changed from Tamar Shacked to Milind Changire
Milind, PTAL. FWIW - https://github.com/ceph/ceph/pull/47504 fixes a similar issue for RGW.
Updated by Milind Changire over 1 year ago
This doesn't crash on my local ubuntu focal vstart cluster.
The stack trace points to a boost::lexical_cast<>
Hypothesis:
I'm not sure if boost::lexical_cast<> has any runtime requirements that may not match with those on the build environment.
Hence the crash.
Updated by Venky Shankar over 1 year ago
Milind Changire wrote:
This doesn't crash on my local ubuntu focal vstart cluster.
The stack trace points to a boost::lexical_cast<>Hypothesis:
I'm not sure if boost::lexical_cast<> has any runtime requirements that may not match with those on the build environment.
Hence the crash.
Its not reproducible always. Did you check what changes were done in https://github.com/ceph/ceph/pull/47504 to mitigate this crash in rgw?
Updated by Milind Changire over 1 year ago
Venky Shankar wrote:
Milind Changire wrote:
This doesn't crash on my local ubuntu focal vstart cluster.
The stack trace points to a boost::lexical_cast<>Hypothesis:
I'm not sure if boost::lexical_cast<> has any runtime requirements that may not match with those on the build environment.
Hence the crash.Its not reproducible always. Did you check what changes were done in https://github.com/ceph/ceph/pull/47504 to mitigate this crash in rgw?
The rgw solution is to stop building radosgw as a shared library and instead build an executable binary.
There were also some references on the internet which discussed about avoiding linking against static libstdc++ when building a shared library.
Nothing conclusive so far. If the problem was indeed about linking a shared library against static libstdc++, then the problem should've been 100% reproducible in a vstart cluster as well.
Updated by Venky Shankar over 1 year ago
Milind Changire wrote:
Venky Shankar wrote:
Milind Changire wrote:
This doesn't crash on my local ubuntu focal vstart cluster.
The stack trace points to a boost::lexical_cast<>Hypothesis:
I'm not sure if boost::lexical_cast<> has any runtime requirements that may not match with those on the build environment.
Hence the crash.Its not reproducible always. Did you check what changes were done in https://github.com/ceph/ceph/pull/47504 to mitigate this crash in rgw?
The rgw solution is to stop building radosgw as a shared library and instead build an executable binary.
There were also some references on the internet which discussed about avoiding linking against static libstdc++ when building a shared library.
Nothing conclusive so far. If the problem was indeed about linking a shared library against static libstdc++, then the problem should've been 100% reproducible in a vstart cluster as well.
Probably depends on the version of libstdc++ I guess. Do you see the version in the logs or maybe another linked (copy) version?
Updated by Venky Shankar over 1 year ago
Seeing this in me recent run - https://pulpito.ceph.com/vshankar-2023-01-25_07:57:32-fs-wip-vshankar-testing-20230125.055346-testing-default-smithi/7138090/
file ./remote/smithi148/coredump/1675013313.79003.core ./remote/smithi148/coredump/1675013313.79003.core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_libcephfs_reclaim', real uid: 1000, effective uid: 1000, real gid: 1267, effective gid: 1267, execfn: '/usr/bin/ceph_test_libcephfs_reclaim', platform: 'x86_64'
Didn't see this segfault in my last run, but, I've been doing fewer runs due to teuthology infra issues.
Milind, Did we RCA this?
Updated by Venky Shankar about 1 year ago
Rishabh Dave wrote:
This shows up once in a while. We didn't RCA this, did we?
Updated by Milind Changire about 1 year ago
Venky Shankar wrote:
Rishabh Dave wrote:
This shows up once in a while. We didn't RCA this, did we?
No RCA yet.
Updated by Venky Shankar 10 months ago
Updated by Laura Flores 10 months ago
- Has duplicate Bug #62228: "Segmentation fault" (['libcephfs/test.sh']) in smoke on reef added
Updated by Laura Flores 10 months ago
- Has duplicate deleted (Bug #62228: "Segmentation fault" (['libcephfs/test.sh']) in smoke on reef)
Updated by Laura Flores 10 months ago
- Related to Bug #62228: "Segmentation fault" (['libcephfs/test.sh']) in smoke on reef added
Updated by Laura Flores 10 months ago
- Backport changed from pacific,quincy to pacific,quincy,reef
Updated by Laura Flores 10 months ago
Found an instance of this in the smoke suite and got some more information from the coredump here: https://tracker.ceph.com/issues/62228#note-2
(gdb) bt
#0 0x00007fd656f8243e in std::locale::operator==(std::locale const&) const () from /usr/lib/ceph/libceph-common.so.2
#1 0x00007fd656e45358 in boost::detail::lcast_ret_unsigned<std::char_traits<char>, unsigned long, char>::convert (this=this@entry=0x7ffd1855c530)
at ./obj-x86_64-linux-gnu/boost/include/boost/lexical_cast/detail/lcast_unsigned_converters.hpp:184
#2 0x00007fd656e44594 in boost::detail::lexical_ostream_limited_src<char, std::char_traits<char> >::shr_unsigned<unsigned long> (output=@0x7ffd1855c510: 1, this=<synthetic pointer>)
at ./obj-x86_64-linux-gnu/boost/include/boost/lexical_cast/detail/lcast_unsigned_converters.hpp:167
#3 boost::detail::lexical_ostream_limited_src<char, std::char_traits<char> >::operator>> (output=@0x7ffd1855c510: 1, this=<synthetic pointer>)
at ./obj-x86_64-linux-gnu/boost/include/boost/lexical_cast/detail/converter_lexical_streams.hpp:641
#4 boost::detail::lexical_converter_impl<unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::try_convert (arg="61", arg="61", result=@0x7ffd1855c510: 1)
at ./obj-x86_64-linux-gnu/boost/include/boost/lexical_cast/detail/converter_lexical.hpp:485
#5 boost::conversion::detail::try_lexical_convert<unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (result=@0x7ffd1855c510: 1, arg="61")
at ./obj-x86_64-linux-gnu/boost/include/boost/lexical_cast/try_lexical_convert.hpp:201
#6 boost::lexical_cast<unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (arg="61") at ./obj-x86_64-linux-gnu/boost/include/boost/lexical_cast.hpp:41
#7 librbd::rbd_features_from_string (orig_value="61", err=err@entry=0x7ffd1855c8d0) at ./src/librbd/Features.cc:67
#8 0x00007fd656abd49f in operator() (value=0x7ffd1855cb80, error_message=<optimized out>, __closure=<optimized out>) at ./obj-x86_64-linux-gnu/src/common/options/rbd_options.cc:44
#9 0x00007fd656a14c26 in std::function<int (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)>::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const (__args#1=0x7ffd1855cd60, __args#0=0x7ffd1855cb80, this=<optimized out>)
at /usr/include/c++/11/bits/std_function.h:586
#10 Option::pre_validate (this=<optimized out>, new_value=<optimized out>, err=<optimized out>) at ./src/common/options.cc:93
#11 0x00007fd656a1721d in Option::parse_value (this=0x7fd654ea8420, raw_val=..., out=<optimized out>, error_message=0x7ffd1855cd60, normalized_value=<optimized out>) at ./src/common/options.cc:147
#12 0x00007fd6569e66c2 in md_config_t::_set_val (this=0x5600deb4b510, values=..., observers=..., raw_val=..., opt=..., level=0, error_message=0x7ffd1855cd60) at ./src/common/config.cc:1395
#13 0x00007fd6569e6b37 in md_config_t::set_val_default (this=<optimized out>, values=..., tracker=..., name=..., val="61") at ./src/common/config.cc:273
#14 0x00007fd6569f6b04 in md_config_t::md_config_t (this=this@entry=0x5600deb4b510, values=..., tracker=..., is_daemon=<optimized out>, is_daemon@entry=false) at /usr/include/c++/11/string_view:137
#15 0x00007fd656997d7c in ceph::common::ConfigProxy::ConfigProxy (is_daemon=<optimized out>, this=0x5600deb47c78) at /usr/include/c++/11/bits/stl_tree.h:211
#16 ceph::common::CephContext::CephContext (this=0x5600deb47c70, module_type_=8, options=...) at ./src/common/ceph_context.cc:703
#17 0x00007fd656998e63 in ceph::common::CephContext::CephContext (this=this@entry=0x5600deb47c70, module_type_=<optimized out>, code_env=code_env@entry=CODE_ENVIRONMENT_LIBRARY, init_flags_=init_flags_@entry=0)
at /usr/include/c++/11/bits/std_function.h:239
#18 0x00007fd6569dcd45 in common_preinit (iparams=..., code_env=code_env@entry=CODE_ENVIRONMENT_LIBRARY, flags=flags@entry=0) at ./src/common/common_init.cc:38
#19 0x00007fd657688ad9 in ceph_create (cmount=0x7ffd185670e0, id=0x0) at ./src/libcephfs.cc:421
#20 0x00005600dd1c891a in update_root_mode () at ./src/test/libcephfs/reclaim.cc:151
#21 main (argc=<optimized out>, argv=0x7ffd18567288) at ./src/test/libcephfs/reclaim.cc:151
Updated by Laura Flores 10 months ago
- Related to Bug #57050: Crash on startup of radosgw in librbd::rbd_features_from_string() added
Updated by Laura Flores 10 months ago
A similar issue was reported and solved in https://tracker.ceph.com/issues/57050. A note from Casey:
What i recall is that it was specific to WITH_STATIC_LIBSTDCXX which we only use for ubuntu focal.
Linking the standard library multiple times is problematic. I guess std::locale crashes there because it depends on global state, but different shared libraries would have different globals.
For rgw, I reverted a commit that moved rgw's global_init stuff into a shared library.
Perhaps something similar needs to happen for cephfs. Looking back at reported instances, all have happened on ubuntu focal.
Updated by Laura Flores 10 months ago
Here's a job where the test passed on ubuntu. Could provide some clues to what changed:
https://pulpito.ceph.com/yuriw-2023-03-24_16:24:00-smoke-quincy-release-distro-default-smithi/7219355/
(sha1=23eb3b2f0fc65087846571af4e15146a980fc03d)
Updated by Laura Flores 10 months ago
Possibly more helpful, here's the last instance of it passing on ubuntu 20.04 main:
https://pulpito.ceph.com/teuthology-2022-07-14_05:00:05-smoke-main-distro-default-smithi/6929877/
And the first instance of it failing on ubuntu 20.04 main:
https://pulpito.ceph.com/teuthology-2022-07-17_05:00:02-smoke-main-distro-default-smithi/6933658/
Could anything here have caused it?
$ git log --pretty=oneline --no-merges f9bd8922d75a454b4a0ace5a85637213e5ee5700..23eace17e0dc2a432b8532c635906af0ce79391e
2de0574382e2c1c63f20745d6870ac7f82b27b9f qa/tasks: rename persistent write log cache trash task
fddb7dd2b038d52489ec74aefa03cce5dc435e77 (origin/ci-fix-mib, ci/ci-fix-mib) ceph.spec.in: fix path for mib file and properly mark in %files
e589d0d35c1a1b2dc5fbee8bb8aae45ec11a9881 crimson/os/seastore: simplify backref cache
29c44a3bd6919f7fc23083d75d22e84c8b51e1c1 (zdover23/wip-doc-2022-07-15-start-hw-recs-cleanup-1) doc/start: update hardware recs
66bea86ab447b2db8a948c7913dc6c7a4a995c31 (origin/snapshot_key_conversion, ci/snapshot_key_conversion) osd/SnapMapper: fix pacific legacy key conversion and introduce test
d4a39cd046b93cb7bb5b7ce0311139d9f6552802 doc/cephadm: add note about OSDs being recreated to OSD removal section
80964fb5204ab18208d1033c927a620448802451 mgr/dashboard: fix rbdconfiguration init type
965005e0789e566ccadce7a326b0e197ab8d7f5f mgr/dashboard: prevent alert redirect
6f5ac1274bce02b1582be92248af7bce892a28b8 cephadm/box: Choose between docker or podman with --engine
ee9195d09efa6773a1abe2390de9d29543228c81 crimson/os/seastore: fix bug of Transaction::is_retired
0eab8de3c017d8318bd6c846991bb3f7c51fa97d qa/tasks: add thrash test for persistent write log cache
08c980ede37f1d2eba08b43ce624994b939a3ca3 mgr/dashboard: fix api tests + lint
23615b3e1b0d479595929d6483856afce67502d2 mgr/dashboard: cleanups
5a0ead8cc88fd677d882391bddcff5b474a63a9b mgr/dashboard: RbdConfiguration.list reuse image context.
c01ce41266aeec229d60c302b4da8ed6fac6f2d0 mgr/dashboard: change sort char to '+' and page icons
90869173ee4cbe2d9e24e354608995e04793bc7d mgr/dashboard: fix linting errors
3c308804d5a7a5e96842e3234edbf6ad42509e93 mgr/dashboard: add rbd list sorting support
5386dd1cb3195e7bc2dd80dafa1f0d29e8e0eb9e mgr/dashboard: update rbd list api version to 2
c832464998ed3ed34af3bda37056dfdc1729f4a6 mgr/dashboard: debounce reload data with serverSide
68ef0b83e9b0ce3ccf854eff9cbf599a263b3e7e mgr/dashboard: add rbd list search and disable sorting
08b6f22807c14967fb49495ca9f18124d495b19b mgr/dashboard: fix rbd list tests
a27eea3e8c834eb6e148aeef72a5e8522d81ef97 mgr/dashboard: fix frontend unit tests
4060061014506ed1edd32936f54ed46d489c0f11 mgr/dashboard: if null tablecontext then create empty one
27f40514ebc87a05144989be9aa1abb700f49566 mgr/dashboard: more linting
78523ec030621be548c4e3ecb8802d0b5c7bd8e8 mgr/dashboard: fix tests and iscsi rbd list
dbf2552ca57d90f6551553d25ca493375eb1e700 mgr/dashboard: bind serverSide to externalPaging
efd9a8ba5c9194e89b95aae021f62f3e164f3581 mgr/dashboard: rbd pagination poc
ed2c0247f9d4e81283e4837c90792a717ca54bc8 examples/boto3: Updating Python examples to work with Python 3
0e303cffdfb342289211f059fde25f679ae37ab9 cephadm: update LATEST_STABLE_RELEASE
eb94cda7bce38d11b9000fe93635f33bf2b3ff12 cmake: add ndctl and daxctl to build pmdk
b6b24852a6d1c6d8e785e15661f3bf29d662417f cmake: add findndctl and finddaxctl function
7fd52c0669e93fb6491ef11043d9c958c2d81b76 .github/labeler.yml: add core label to some tools
d44c0f696bac30efb79a9a99a897b4c9e994409d tools/crimson/perf_crimson_msgr: correct default addr to msgr v2 version
4cdcf7f1559e0e1453f85253fe5c4bc5ef64a93d tools/crimson/perf_crimson_msgr: init ConfigPorxy when pref_crimson_msgr start running
a581509381ba84b49c906a1fe440ca3ddcab418c rbd-mirror: remove bogus completed_non_primary_snapshots_exist check
13aa47ae2ee89babca7571165ac88eaa665d2fc0 test/rbd-mirror: add last_copied_object_number == 0 coverage
d4e6763a19fb91cd0b25c835ed9f9ccbfe70c07f cephadm/box: Rootless podman box implementation
deb7412d6f4de155bc6843f51c792b7501d2ff97 ceph.spec.in: review changes for mibs package
3a7c8c602282c8d9e95b793985237f7b3df50db7 package SNMP MIB file as an rpm
52dc04ca8a5de0626b334c7544f806b7b7dbd79a mgr/dashboard: rbd striping setting pre-population and pop-over
ae8a0dcebd21ede42aad656ba0937058de58b0bc src/stop.sh: add `sudo` to `cephadm ls` command
37aafb9170f677766996dbec584917cc6b8509b9 src/stop.sh: fixup ceph-mds daemon count
Updated by Venky Shankar 10 months ago
- Assignee changed from Milind Changire to Venky Shankar
- Priority changed from Normal to High
Thanks for the details, Laura. I'll have a look.
Updated by Venky Shankar 10 months ago
Laura Flores wrote:
Possibly more helpful, here's the last instance of it passing on ubuntu 20.04 main:
https://pulpito.ceph.com/teuthology-2022-07-14_05:00:05-smoke-main-distro-default-smithi/6929877/And the first instance of it failing on ubuntu 20.04 main:
https://pulpito.ceph.com/teuthology-2022-07-17_05:00:02-smoke-main-distro-default-smithi/6933658/Could anything here have caused it?
I can't zero-in on a particular commit that could have triggered this.
Updated by Venky Shankar 10 months ago
The difference I can see in src/test/libcephfs/CMakeLists.txt is
for, say, test.cc:
target_link_libraries(ceph_test_libcephfs ceph-common cephfs ${UNITTEST_LIBS} ${EXTRALIBS} ${CMAKE_DL_LIBS} )
But, for reclaim.cc:
target_link_libraries(ceph_test_libcephfs_reclaim cephfs ${UNITTEST_LIBS} ${EXTRALIBS} ${CMAKE_DL_LIBS} )
Where an explicit linkage to ceph-common is missing. The crash backtrace has `common_preinit` as its origination which is a part of ceph-common, so, I think this crash has to do with another version of the lib that's getting linked somehow. Pretty weird.
Updated by Venky Shankar 10 months ago
- Status changed from Triaged to Fix Under Review
- Pull request ID set to 52754
Updated by Venky Shankar 10 months ago
- Status changed from Fix Under Review to Pending Backport
- Target version changed from v18.0.0 to v19.0.0
Updated by Backport Bot 10 months ago
- Copied to Backport #62287: reef: ceph_test_libcephfs_reclaim crashes during test added
Updated by Backport Bot 10 months ago
- Copied to Backport #62288: pacific: ceph_test_libcephfs_reclaim crashes during test added
Updated by Backport Bot 10 months ago
- Copied to Backport #62289: quincy: ceph_test_libcephfs_reclaim crashes during test added
Updated by Venky Shankar 6 months ago
- Status changed from Pending Backport to Rejected
Updated by Laura Flores 2 months ago
- Translation missing: en.field_tag_list set to test-failure
/a/yuriw-2024-03-05_15:31:54-smoke-reef-release-distro-default-smithi/7582350
Updated by Rishabh Dave about 2 months ago
- Status changed from Rejected to Resolved
This ticket was incorrectly marked as rejected. The patch for this ticket was merged - https://github.com/ceph/ceph/pull/52754#event-9990239615. And the Pacific backport for it was rejected, see - https://github.com/ceph/ceph/pull/53648#issuecomment-1802128014, https://tracker.ceph.com/issues/62288#note-3.
Since this issue has been fixed, marking this as resolved.