Autor Téma: FireDAC segmentation fault v linuxu (ubuntu 16.04 LTS)  (Přečteno 462 krát)

Offline maca2me

  • Nováček
  • *
  • Příspěvků: 24
  • Karma: 1
    • Verze Delphi: 10.2
Ahoj,

narazili jsme na problém při použití komponent FireDAC (FDManager, FDEventAlerter, FDQuery...) v multivláknové aplikaci (cca 20*) v linuxu (Ubuntu 16.04 LTS). SW je v náhodných časových intervalech (1 h, 10 h, 2 dny) ukončen signálem SIGSEGV. Podle výpisu GDB se v drtivé většině jednalo o volání funkcí .Open, .Close, .Create na těchto komponentách v náhodných místech. V případě, že je program spuštěn jako daemon, chyba se vyskytne častěji ale spadne to i jako obyč. program.

Nesetkal jste se někdo s podobným problémem? Případně nevíte, na co se zaměřit?

Díky moc
Neexistují hloupé otázky ale jen hloupé odpovědi.

Online Delfin

  • Guru
  • *****
  • Příspěvků: 1201
  • Karma: 53
  • SW konzultant
    • Verze Delphi: 2009, Tokyo
Re:FireDAC segmentation fault v linuxu (ubuntu 16.04 LTS)
« Odpověď #1 kdy: 10-08-2018, 07:45:04 »
Případně nevíte, na co se zaměřit?

Na ten GDB log :)? Je naprosto nemozne radit cokoli konkretniho. Moznosti jak FireDAC zmrzacit jsou tisice. Zmrzacit ho vsak zpusobem ze na AV skonci volani konstruktoru (jakeho objektu?) zni dost kriticky.
I'm a soldier, so don't panic! I know the underground! I like WTFPL license! No more Google, go duck, go!

Offline pf1957

  • Padawan
  • ******
  • Příspěvků: 2269
  • Karma: 122
    • Verze Delphi: D2007, XE3, DX10
Re:FireDAC segmentation fault v linuxu (ubuntu 16.04 LTS)
« Odpověď #2 kdy: 10-08-2018, 09:07:20 »
se v drtivé většině jednalo o volání funkcí .Open, .Close, .Create na těchto komponentách v náhodných místech.
Takova kontrolni otazka:
- ma kazdy thread vlastni DB connectivitu (obecne pravidlo pro praci s DB v multitreaded prostredi)?
- Kdyz instanci neceho nepotrebujete, zavirat a destruujete?

Offline maca2me

  • Nováček
  • *
  • Příspěvků: 24
  • Karma: 1
    • Verze Delphi: 10.2
Re:FireDAC segmentation fault v linuxu (ubuntu 16.04 LTS)
« Odpověď #3 kdy: 10-08-2018, 10:08:51 »
Díky za reakce.

Volání .Open .Close a .Create bylo převážně na objektech FireDAC - hlavně query. Každý objekt FireDAC je ukončen a uvolněn pomocí FDFreeAndNil. Jinak rostla paměť (při použití jen .Free).

Používáme FDManager pro spravování poolu s connections. Tedy pro každé vlákno máme vlastní connection. Využíváme FDEventAlerter pro události z Firebird DB s nastavením synchronize false. Tím předpokládáme, že každý event je vykonáván vlastním vláknem na pozadí.

Chyby se vyskytují naprosto nahodile. Typicky se nějaká funkce zavolá 1000x a po 1001 to spadne. Nejspíš to vypadá na nějaký problém při alokaci paměti, nebo problém s vlákny apod. Rad studio 10.2 je první verze pro linux, tak doufáme, že není problém dokonce někde v tomto.

Vybraný GDB Log:

Kód: [Vybrat]

Thread 22 "HavisAppLinux" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffccff9700 (LWP 1351)]
0x0000000000449c96 in System::TMonitor::Enter(unsigned int) (Self=0x1,
    Timeout=4294967295) at System.pas:18572
18572   System.pas: No such file or directory.
(gdb) bt
#0  0x0000000000449c96 in System::TMonitor::Enter(unsigned int) (Self=0x1,
    Timeout=4294967295) at System.pas:18572
#1  0x0000000000455e41 in System::TInstItem::Lock() (Self=0x7fffd412f5d0)
    at System.pas:37556
#2  0x0000000000455ee5 in System::TInstItem::RegisterRef(System::StaticArray<Sys                                                                                                                                                             tem::DynamicArray<void**>, 31>&, void*) (Self=0x7fffd412f5d0, Buckets=...,
    Address=0x7fffd4155c10) at System.pas:37587
#3  0x00000000004558d8 in System::TInstHashMap::RegisterWeakMethodRef(void*, voi                                                                                                                                                             d*) (Self=0xec8260, Address=0x7fffd4155c10, Instance=0x7fffd4155bf0)
    at System.pas:37901
#4  0x0000000000456504 in System::RegisterWeakMethodRef(void*, System::TObject*)                                                                                                                                                              (Address=0x7fffd4155c10, Instance=0x7fffd4155bf0) at System.pas:37997
#5  0x0000000000456584 in System::_AsgClosureObj(System::TMethod&, void const*,                                                                                                                                                              void const*) (Dest=..., Obj=0x7fffd4155bf0, MethodAddr=0x5d35b0)
    at System.pas:38132
#6  0x000000000058ec2a in System::Generics::Collections::TList__1<System::Classe                                                                                                                                                             s::TCollectionItem*>::TList__1(System::DelphiInterface<System::Generics::Default                                                                                                                                                             s::IComparer__1<System::Classes::TCollectionItem*> >) (Self=0x7fffd4155bf0,
    .=false, AComparer=0xcd3880) at System.Generics.Collections.pas:4137
#7  0x000000000058ea45 in System::Generics::Collections::TList__1<System::Classe                                                                                                                                                             s::TCollectionItem*>::TList__1() (Self=0x7fffd4155bf0, .=true)
    at System.Generics.Collections.pas:4131
#8  0x0000000000560669 in System::Classes::TCollection::TCollection(System::Clas                                                                                                                                                             ---Type <return> to continue, or q <return> to quit---
ses::TCollectionItemClass) (Self=0x7fffd4159810, .=false, ItemClass=0xdab558) at System.Classes.pas:5826
#9  0x00000000006deff7 in Firedac::Stan::Param::TFDMacros::TFDMacros(System::Classes::TPersistent* (* __closure)()) (Self=0x7fffd4159810, .=true, AGetOwner=...) at FireDAC.Stan.Param.pas:927
#10 0x00000000007c6d5f in Firedac::Phys::TFDPhysCommand::SetMacros(Firedac::Stan::Param::TFDMacros*) (Self=0x7fffd416a2e0, AValue=0x0) at FireDAC.Phys.pas:6310
#11 0x0000000000854bcd in Firedac::Comp::Client::TFDCustomCommand::InternalUnprepare() (Self=0x7fffd4016850) at FireDAC.Comp.Client.pas:6449
#12 0x000000000085a240 in Firedac::Comp::Client::TFDCustomCommand::SetPrepared(bool) (Self=0x7fffd4016850, AValue=false) at FireDAC.Comp.Client.pas:6557
#13 0x00000000008576fa in Firedac::Comp::Client::TFDCustomCommand::Unprepare() (Self=0x7fffd4016850) at FireDAC.Comp.Client.pas:6608
#14 0x0000000000865c25 in Firedac::Comp::Client::TFDAdaptedDataSet::DoUnprepareSource() (Self=0x7fffd4010d40) at FireDAC.Comp.Client.pas:9889
#15 0x00000000007ff103 in Firedac::Comp::Dataset::TFDDataSet::ReleaseBase(bool) (Self=0x7fffd4010d40, AOffline=true) at FireDAC.Comp.DataSet.pas:3188
#16 0x0000000000863f02 in Firedac::Comp::Client::TFDAdaptedDataSet::ReleaseBase(bool) (Self=0x7fffd4010d40, AOffline=true) at FireDAC.Comp.Client.pas:9481
#17 0x000000000086c117 in Firedac::Comp::Client::TFDRdbmsDataSet::ReleaseBase(bool) (Self=0x7fffd4010d40, AOffline=true) at FireDAC.Comp.Client.pas:11079
#18 0x000000000080983e in Firedac::Comp::Dataset::TFDDataSet::Offline() (Self=0x7fffd4010d40) at FireDAC.Comp.DataSet.pas:3199
#19 0x00000000008508f9 in Firedac::Comp::Client::TFDCustomTransaction::HandleDisconnectCommands(bool (*)(System::TObject*), Firedac::Phys::Intf::TFDPhysDisconnectMode) (Self=0x7fffd40074c0, AFilter=0x0, AMode=dmOffline)
    at FireDAC.Comp.Client.pas:5749
#20 0x00000000007d5c75 in Firedac::Phys::TFDPhysTransaction::DisconnectCommands(bool (*)(System::TObject*), Firedac::Phys::Intf::TFDPhysDisconnectMode) (Self=0x7fffd400c1e0, AFilter=0x0, AMode=dmOffline) at FireDAC.Phys.pas:4487
#21 0x00000000008de584 in Firedac::Phys::Ibbase::TFDPhysIBTransactionBase::InternalCommit(unsigned long) (Self=0x7fffd400c1e0, AID=4693) at FireDAC.Phys.IBBase.pas:1944
#22 0x00000000007bed87 in Firedac::Phys::TFDPhysTransaction::Commit() (Self=0x7fffccff7838) at FireDAC.Phys.pas:4858
#23 0x00000000007d5fb3 in Firedac::Phys::TFDPhysTransaction::Stop(bool) (Self=0x7fffd400c1e0, ASuccess=true) at FireDAC.Phys.pas:4519
#24 0x00000000007d62a5 in Firedac::Phys::TFDPhysTransaction::CheckStoping(bool, bool, bool) (Self=0x7fffd400c1e0, AAllowUnprepare=true, AForce=false, ASuccess=true) at FireDAC.Phys.pas:4563
#25 0x00000000008df872 in Firedac::Phys::Ibbase::TFDPhysIBTransactionBase::InternalNotify(Firedac::Phys::TFDPhysTxNotification, Firedac::Phys::TFDPhysCommand*) (Self=0x7fffd400c1e0, ANotification=cpAfterCmdExecuteSuccess,
    ACommandObj=0x7fffd416a2e0) at FireDAC.Phys.IBBase.pas:2003
#26 0x00000000007d6393 in Firedac::Phys::TFDPhysTransaction::Notify(Firedac::Phys::TFDPhysTxNotification, Firedac::Phys::TFDPhysCommand*) (Self=0x7fffd400c1e0, ANotification=cpAfterCmdExecuteSuccess, ACommandObj=0x7fffd416a2e0)
    at FireDAC.Phys.pas:4591
#27 0x00000000007e526f in Firedac::Phys::TFDPhysCommand::ExecuteBase(int, int) (Self=0x7fffccff7ee8, ATimes=-855671048, AOffset=-855671044) at FireDAC.Phys.pas:8515
#28 0x00000000007e4607 in Firedac::Phys::TFDPhysCommandAsyncExecute::Execute() (Self=0x7fffd40627e0) at FireDAC.Phys.pas:8056
#29 0x000000000077d38a in Firedac::Stan::Async::TFDStanAsyncExecutor::ExecuteOperation(bool) (Self=0x7fffd416d2b0, AStoreException=false) at FireDAC.Stan.Async.pas:173
#30 0x000000000077c87e in Firedac::Stan::Async::TFDStanAsyncExecutor::Run() (Self=0x7fffd416d2b0) at FireDAC.Stan.Async.pas:268
#31 0x00000000007d9ad3 in Firedac::Phys::TFDPhysCommand::ExecuteTask(System::DelphiInterface<Firedac::Stan::Intf::IFDStanAsyncOperation>, System::DelphiInterface<Firedac::Stan::Intf::IFDStanAsyncHandler>, bool) (Self=0x7fffd416a2e0,
    AOperation=0x7fffd4062800, AHandler=0x7fffd4016a60, ABlocked=false) at FireDAC.Phys.pas:6608
#32 0x00000000007c84ea in Firedac::Phys::TFDPhysCommand::Execute(int, int, bool) (Self=0x7fffd416a2e0, ATimes=0, AOffset=0, ABlocked=false) at FireDAC.Phys.pas:8537
#33 0x0000000000855b4e in Firedac::Comp::Client::TFDCustomCommand::InternalExecute(int, int, bool) (Self=0x7fffd4016850, ATimes=0, AOffset=0, ABlocked=false) at FireDAC.Comp.Client.pas:6852
#34 0x0000000000857a18 in Firedac::Comp::Client::TFDCustomCommand::Execute(int, int, bool) (Self=0x7fffd4016850, ATimes=0, AOffset=0, ABlocked=false) at FireDAC.Comp.Client.pas:6875
#35 0x00000000008666a7 in Firedac::Comp::Client::TFDAdaptedDataSet::DoExecuteSource(int, int) (Self=0x7fffd4010d40, ATimes=0, AOffset=0) at FireDAC.Comp.Client.pas:10078
#36 0x0000000000804707 in Firedac::Comp::Dataset::TFDDataSet::Execute(int, int) (Self=0x7fffd4010d40, ATimes=0, AOffset=0) at FireDAC.Comp.DataSet.pas:3133
#37 0x000000000086f9e2 in Firedac::Comp::Client::TFDCustomQuery::ExecSQL() (Self=0x7fffd4010d40) at FireDAC.Comp.Client.pas:11496
#38 0x0000000000a2868e in Uautomatthread::TAutomatThread::Execute() (Self=0x10252b0) at uAutomatThread.pas:161
#39 0x00000000005cd517 in System::Classes::ThreadProc(System::Classes::TThread*) (Thread=0x10252b0) at System.Classes.pas:14900
#40 0x000000000044d370 in System::ThreadWrapper(void*) (Parameter=0xf61350) at System.pas:24287
#41 0x00007ffff73dd6ba in start_thread (arg=0x7fffccff9700) at pthread_create.c:333
#42 0x00007ffff78fe41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb)

A ještě druhý - přímo na FD open.
Kód: [Vybrat]
Akce zprávyHTMLText    Odpovědět odesílateli Odpovědět všem Předat zprávu   Otevřít v novém okně
Předmět: gdb
Fotografie kontaktu
Odesílatel borek@starmon.cz Datum Dnes 10:21
Tělo zprávy
Thread 5 "HavisAppLinux" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffee4e3700 (LWP 1631)]
0x0000000000449c96 in System::TMonitor::Enter(unsigned int) (Self=0x4, Timeout=4294967295) at System.pas:18572
18572   System.pas: No such file or directory.
(gdb) bt
#0  0x0000000000449c96 in System::TMonitor::Enter(unsigned int) (Self=0x4, Timeout=4294967295) at System.pas:18572
#1  0x0000000000455e41 in System::TInstItem::Lock() (Self=0x7fffb0119be0) at System.pas:37556
#2  0x0000000000455ee5 in System::TInstItem::RegisterRef(System::StaticArray<System::DynamicArray<void**>, 31>&, void*) (Self=0x7fffb0119be0, Buckets=..., Address=0x7fffe4480750) at System.pas:37587
#3  0x00000000004558d8 in System::TInstHashMap::RegisterWeakMethodRef(void*, void*) (Self=0xec8260,
    Address=0x7fffe4480750, Instance=0x7fffe4480730) at System.pas:37901
#4  0x0000000000456504 in System::RegisterWeakMethodRef(void*, System::TObject*) (Address=0x7fffe4480750,
    Instance=0x7fffe4480730) at System.pas:37997
#5  0x0000000000456584 in System::_AsgClosureObj(System::TMethod&, void const*, void const*) (Dest=...,
    Obj=0x7fffe4480730, MethodAddr=0x5d35b0) at System.pas:38132
#6  0x000000000058ec2a in System::Generics::Collections::TList__1<System::Classes::TCollectionItem*>::TList__1(System::DelphiInterface<System::Generics::Defaults::IComparer__1<System::Classes::TCollectionItem*> >) (
    Self=0x7fffe4480730, .=false, AComparer=0xcd3880) at System.Generics.Collections.pas:4137
#7  0x000000000058ea45 in System::Generics::Collections::TList__1<System::Classes::TCollectionItem*>::TList__1() (
    Self=0x7fffe4480730, .=true) at System.Generics.Collections.pas:4131
#8  0x0000000000560669 in System::Classes::TCollection::TCollection(System::Classes::TCollectionItemClass) (
    Self=0x7fffe447d160, .=false, ItemClass=0xdab558) at System.Classes.pas:5826
#9  0x00000000006deff7 in Firedac::Stan::Param::TFDMacros::TFDMacros(System::Classes::TPersistent* (* __closure)())
    (Self=0x7fffe447d160, .=true, AGetOwner=...) at FireDAC.Stan.Param.pas:927
#10 0x00000000007cbb0d in Firedac::Phys::TFDPhysCommand::TFDPhysCommand(Firedac::Phys::TFDPhysConnection*) (
    Self=0x7fffe44822a0, .=true, AConnection=0x7fffe400cd40) at FireDAC.Phys.pas:5999
#11 0x00000000008f1c69 in Firedac::Phys::Fb::TFDPhysFBConnection::InternalCreateCommand() (Self=0x7fffe400cd40)
    at FireDAC.Phys.FB.pas:446
#12 0x00000000007b8303 in Firedac::Phys::TFDPhysConnection::CreateCommand(System::DelphiInterface<Firedac::Phys::Intf::IFDPhysCommand>&) (Self=0x7fffee4e03b0, ACmd=@0x7fffe447ee90: 0x0) at FireDAC.Phys.pas:3116
#13 0x0000000000854367 in Firedac::Comp::Client::TFDCustomCommand::InternalCreateCommandIntf() (Self=0x7fffe447ee10)
    at FireDAC.Comp.Client.pas:6333
#14 0x0000000000859bcd in Firedac::Comp::Client::TFDCustomCommand::SetPrepared(bool) (Self=0x7fffe447ee10,
    AValue=true) at FireDAC.Comp.Client.pas:6508
#15 0x00000000008576da in Firedac::Comp::Client::TFDCustomCommand::Prepare(System::UnicodeString) (
    Self=0x7fffe447ee10,
    ACommandText=<unknown type in /bin/HAVIS/AppServer/HavisAppLinux, CU 0x61864b, DIE 0x6198c2>)
    at FireDAC.Comp.Client.pas:6602
#16 0x0000000000865a64 in Firedac::Comp::Client::TFDAdaptedDataSet::DoPrepareSource() (Self=0x7fffe4474260)
    at FireDAC.Comp.Client.pas:9867
#17 0x00000000007f876e in Firedac::Comp::Dataset::TFDDataSet::OpenCursor(bool) (Self=0x7fffe4474260,
    InfoQuery=false) at FireDAC.Comp.DataSet.pas:2897
#18 0x000000000086bc9e in Firedac::Comp::Client::TFDRdbmsDataSet::OpenCursor(bool) (Self=0x7fffe4474260,
    InfoQuery=false) at FireDAC.Comp.Client.pas:11142
#19 0x00000000006946c9 in Data::Db::TDataSet::SetActive(bool) (Self=0x7fffe4474260, Value=true) at Data.DB.pas:12659
#20 0x00000000007f8ec1 in Firedac::Comp::Dataset::TFDDataSet::SetActive(bool) (Self=0x7fffe4474260, AValue=true)
    at FireDAC.Comp.DataSet.pas:1606
#21 0x000000000069905e in Data::Db::TDataSet::Open() (Self=0x7fffe4474260) at Data.DB.pas:12615
#22 0x000000000093bfc1 in Udb::getStationStates(Firedac::Comp::Client::TFDQuery*, System::UnicodeString) (
    aQuery=0x7fffe4474260, asId=<unknown type in /bin/HAVIS/AppServer/HavisAppLinux, CU 0x7bf3a9, DIE 0x808d80>)
    at uDB.pas:2247
#23 0x000000000093818d in Udb::getChangedData(Firedac::Comp::Client::TFDQuery*, System::UnicodeString) (
    aQuery=0x7fffe4474260, aId=<unknown type in /bin/HAVIS/AppServer/HavisAppLinux, CU 0x7bf3a9, DIE 0x808c56>)
    at uDB.pas:1973
#24 0x0000000000a54d08 in Hgeneral::TMyAlert::updateTrain() (Self=0xf2c0b0) at hGeneral.pas:1628
#25 0x0000000000a54529 in Hgeneral::TMyAlert::doAlert(Firedac::Comp::Client::TFDCustomEventAlerter*, System::UnicodeString, System::Variant const&) (Self=0xf2c0b0, ASender=0xf9d910,
    AEventName=<unknown type in /bin/HAVIS/AppServer/HavisAppLinux, CU 0xe2f17b, DIE 0xe3042a>, AArgument=...)
    at hGeneral.pas:1579
#26 0x0000000000851bb2 in Firedac::Comp::Client::TFDCustomEventAlerter::HandleEvent(System::UnicodeString, System::Variant const&) (Self=0xf9d910,
    AEventName=<unknown type in /bin/HAVIS/AppServer/HavisAppLinux, CU 0x61864b, DIE 0x6198c2>, AArgument=...)
    at FireDAC.Comp.Client.pas:5933
#27 0x00000000007c4ace in Firedac::Phys::TFDPhysEventAlerter::InternalHandleEvent(System::UnicodeString, System::Variant const&) (Self=0xfa26f0,
    AEventName=<unknown type in /bin/HAVIS/AppServer/HavisAppLinux, CU 0x53c6bf, DIE 0x53d936>, AArgument=...)
---Type <return> to continue, or q <return> to quit---
    at FireDAC.Phys.pas:5803
#28 0x00000000008dffb2 in Firedac::Phys::Ibbase::TFDPhysIBEventAlerterBase::InternalHandle(Firedac::Phys::TFDPhysEventMessage*) (Self=0xfa26f0, AEventMessage=0x7fffd8001490) at FireDAC.Phys.IBBase.pas:2093
#29 0x00000000007d7155 in Firedac::Phys::TFDPhysEventMessage::BasePerform() (Self=0x7fffd8001490)
    at FireDAC.Phys.pas:5279
#30 0x00000000007c232c in Firedac::Phys::TFDPhysEventMessage::Perform(Firedac::Stan::Util::TFDThread*) (
    Self=0x7fffd8001490, AThread=0xff0740) at FireDAC.Phys.pas:5290
#31 0x00000000006c637d in Firedac::Stan::Util::TFDThread::Execute() (Self=0x7fffee4e2df8)
    at FireDAC.Stan.Util.pas:726
#32 0x00000000005cd517 in System::Classes::ThreadProc(System::Classes::TThread*) (Thread=0xff0740)
    at System.Classes.pas:14900
#33 0x000000000044d370 in System::ThreadWrapper(void*) (Parameter=0xfce8b0) at System.pas:24287
#34 0x00007ffff73dd6ba in start_thread (arg=0x7fffee4e3700) at pthread_create.c:333
#35 0x00007ffff78fe41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 

Ing. Filip Borek

Programátor

STARMON, s.r.o., Průmyslová 1880, 565 01 Choceň

Pracoviště: Bratří Hubálků 161, 560 02 Česká Třebová

mobil: +420 601 102 967

borek@starmon.cz, www.starmon.cz

Starmon_LOGO

Podrobné informace o zpracování osobních údajů naleznete na www.starmon.cz

 


« Poslední změna: 10-08-2018, 10:22:40 od maca2me »
Neexistují hloupé otázky ale jen hloupé odpovědi.

Offline pf1957

  • Padawan
  • ******
  • Příspěvků: 2269
  • Karma: 122
    • Verze Delphi: D2007, XE3, DX10
Re:FireDAC segmentation fault v linuxu (ubuntu 16.04 LTS)
« Odpověď #4 kdy: 10-08-2018, 10:46:27 »
Chyby se vyskytují naprosto nahodile. Typicky se nějaká funkce zavolá 1000x a po 1001 to spadne.
Nejspíš to vypadá na nějaký problém při alokaci paměti, nebo problém s vlákny apod.
No, takhle se typicky projevuji race conditions tj. kdyz se vice threadu/procesu pokousi pristupovat ke sdilenemu prostredku. A chyba vznika nahodile, kdyz se u nejakeho prostredku (pameti, spojeni, souboru atd.) nekoordinovane sejde vice soucasnych pristupu.

Me v tom logu mate, ze se tam objevi chyba "System.pas: No such file or directory", kdyz se to pomoci monitoru pokouselo ziskat pristup buhvi k cemu pri praci s generickym seznamem. Asi bych sel hledat bubakem, co se tam presne deje, kdyz je to OK.


 

S rychlou odpovědí můžete používat BB kódy a emotikony jako v běžném okně pro odpověď, ale daleko rychleji.

Jméno: E-mail:
Ověření:
Kolik je šest plus čtyři (slovem):